Add LittleFS as internal flash filesystem

Adds a LittleFS object which uses the ARMmbed littlefs embedded filesystem,
https://github.com/ARMmbed/littlefs, to enable a new filesystem for onboard
flash utilizing the exact same API as the existing SPIFFS filesystem.

LittleFS is built for low memory systems that are subject to random power
losses, is actively supported by the ARMmbed community, supports directories,
and seems to be much faster in the large-ish read-mostly applications I use.

LittleFS, however, has a larger minimum file allocation unit and does not do
static wear levelling.  This means that for systems that need many little
files (<4K), have small SPIFFS areas (64K), or which have a large static
set of files covering the majority of flash coupled with a frequently
updated set of other files, it may not perform as well.

Simply replace SPIFFS.begin() with LittleFS.begin() in your sketch,
use LittleFS.open in place of SPIFFS.open to open files, and everything
else just works thanks to the magic of @igrr's File base class.

**LITTLEFS FLASH LAYOUT IS INCOMPATIBLE WITH SPIFFS**
Since it is a completely different filesystem, you will need to reformat
your flash (and lose any data therein) to use it. Tools to build the
flash filesystem and upload are at
https://github.com/earlephilhower/arduino-esp8266littlefs-plugin and
https://github.com/earlephilhower/mklittlefs/ .  The mklittlefs tool
is installed as part of the Arduino platform installation, automatically.

The included example shows a contrived read-mostly example and
demonstrates how the same calls work on either SPIFFS.* or LittleFS.*
Host tests are also included as part of CI.

Directories are fully supported in LittleFS. This means that LittleFS
will have a slight difference vs. SPIFFS when you use
LittleFS.openDir()/Dir.next().  On SPIFFS dir.next()
will return all filesystem entries, including ones in "subdirs"
(because in SPIFFS there are no subdirs and "/" is the same as any
other character in a filename).

On LittleFS, dir.next() will only return entries in the directory
specified, not subdirs.  So to list files in "/subdir/..." you need
to actually openDir("/subdir") and use Dir.next() to parse through
just those elements.  The returned filenames also only have the
filename returned, not full paths.  So on a FS with "/a/1", "/a/2"
when you do openDir("/a"); dir.next().getName(); you get "1" and "2"
and not "/a/1" and "/a/2" like in SPIFFS.  This is consistent with
POSIX ideas about reading directories and more natural for a FS.

Most code will not be affected by this, but if you depend on
openDir/Dir.next() you need to be aware of it.

Corresponding ::mkdir, ::rmdir, ::isDirectory, ::isFile,
::openNextFile, and ::rewind methods added to Filesystem objects.
Documentation has been updated with this and other LittleFS information.

Subdirectories are made silently when they do not exist when you
try and create a file in a subdir.  They are silently removed when
the last file in them is deleted.  This is consistent with what
SPIFFS does but is obviously not normal POSIX behavior.  Since there
has never been a "FS.mkdir()" method this is the only way to be
compatible with legacy SPIFFS code.

SPIFFS code has been refactored to pull out common flash_hal_* ops
and placed in its own namespace, like LittleFS.

Author: earlephilhower

.gitmodules

@@ -7,3 +7,6 @@
 [submodule "libraries/SoftwareSerial"]
 	path = libraries/SoftwareSerial
 	url = https://github.com/plerup/espsoftwareserial.git
+[submodule "libraries/LittleFS/lib/littlefs"]
+	path = libraries/LittleFS/lib/littlefs
+	url = https://github.com/ARMmbed/littlefs

README.md

@@ -138,3 +138,5 @@ ESP8266 core files are licensed under LGPL.
 [axTLS](http://axtls.sourceforge.net/) library written by Cameron Rich, built from https://github.com/igrr/axtls-8266, is used in this project. It is distributed under [BSD license](https://github.com/igrr/axtls-8266/blob/master/LICENSE).
 
 [BearSSL](https://bearssl.org) library written by Thomas Pornin, built from https://github.com/earlephilhower/bearssl-esp8266, is used in this project.  It is distributed under the [MIT License](https://bearssl.org/#legal-details).
+
+[LittleFS](https://github.com/ARMmbed/littlefs) library written by ARM Limited and released under the [BSD 3-clause license](https://github.com/ARMmbed/littlefs/blob/master/LICENSE.md).

cores/esp8266/Esp.cpp

@@ -510,14 +510,14 @@ uint32_t EspClass::getSketchSize() {
     return result;
 }
 
-extern "C" uint32_t _SPIFFS_start;
+extern "C" uint32_t _FS_start;
 
 uint32_t EspClass::getFreeSketchSpace() {
 
     uint32_t usedSize = getSketchSize();
     // round one sector up
     uint32_t freeSpaceStart = (usedSize + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
-    uint32_t freeSpaceEnd = (uint32_t)&_SPIFFS_start - 0x40200000;
+    uint32_t freeSpaceEnd = (uint32_t)&_FS_start - 0x40200000;
 
 #ifdef DEBUG_SERIAL
     DEBUG_SERIAL.printf("usedSize=%u freeSpaceStart=%u freeSpaceEnd=%u\r\n", usedSize, freeSpaceStart, freeSpaceEnd);

cores/esp8266/FS.cpp

@@ -121,6 +121,44 @@ const char* File::name() const {
     return _p->name();
 }
 
+const char* File::fullName() const {
+    if (!_p)
+        return nullptr;
+
+    return _p->fullName();
+}
+
+bool File::isFile() const {
+    if (!_p)
+        return false;
+
+    return _p->isFile();
+}
+
+bool File::isDirectory() const {
+    if (!_p)
+        return false;
+
+    return _p->isDirectory();
+}
+
+void File::rewindDirectory() {
+    if (!_fakeDir) {
+        _fakeDir = std::make_shared<Dir>(_baseFS->openDir(fullName()));
+    } else {
+        _fakeDir->rewind();
+   }
+}
+
+File File::openNextFile() {
+    if (!_fakeDir) {
+        _fakeDir = std::make_shared<Dir>(_baseFS->openDir(fullName()));
+    }
+    _fakeDir->next();
+    return _fakeDir->openFile("r");
+}
+
+
 File Dir::openFile(const char* mode) {
     if (!_impl) {
         return File();
@@ -133,7 +171,7 @@ File Dir::openFile(const char* mode) {
         return File();
     }
 
-    return File(_impl->openFile(om, am));
+    return File(_impl->openFile(om, am), _baseFS);
 }
 
 String Dir::fileName() {
@@ -152,6 +190,20 @@ size_t Dir::fileSize() {
     return _impl->fileSize();
 }
 
+bool Dir::isFile() const {
+    if (!_impl)
+        return false;
+
+    return _impl->isFile();
+}
+
+bool Dir::isDirectory() const {
+    if (!_impl)
+        return false;
+
+    return _impl->isDirectory();
+}
+
 bool Dir::next() {
     if (!_impl) {
         return false;
@@ -160,6 +212,14 @@ bool Dir::next() {
     return _impl->next();
 }
 
+bool Dir::rewind() {
+    if (!_impl) {
+        return false;
+    }
+
+    return _impl->rewind();
+}
+
 bool FS::begin() {
     if (!_impl) {
         return false;
@@ -202,8 +262,7 @@ File FS::open(const char* path, const char* mode) {
         DEBUGV("FS::open: invalid mode `%s`\r\n", mode);
         return File();
     }
-
-    return File(_impl->open(path, om, am));
+    return File(_impl->open(path, om, am), this);
 }
 
 bool FS::exists(const char* path) {
@@ -221,7 +280,8 @@ Dir FS::openDir(const char* path) {
     if (!_impl) {
         return Dir();
     }
-    return Dir(_impl->openDir(path));
+    DirImplPtr p = _impl->openDir(path);
+    return Dir(p, this);
 }
 
 Dir FS::openDir(const String& path) {
@@ -239,6 +299,28 @@ bool FS::remove(const String& path) {
     return remove(path.c_str());
 }
 
+bool FS::rmdir(const char* path) {
+    if (!_impl) {
+        return false;
+    }
+    return _impl->rmdir(path);
+}
+
+bool FS::rmdir(const String& path) {
+    return rmdir(path.c_str());
+}
+
+bool FS::mkdir(const char* path) {
+    if (!_impl) {
+        return false;
+    }
+    return _impl->mkdir(path);
+}
+
+bool FS::mkdir(const String& path) {
+    return mkdir(path.c_str());
+}
+
 bool FS::rename(const char* pathFrom, const char* pathTo) {
     if (!_impl) {
         return false;
@@ -251,6 +333,7 @@ bool FS::rename(const String& pathFrom, const String& pathTo) {
 }
 
 
+
 static bool sflags(const char* mode, OpenMode& om, AccessMode& am) {
     switch (mode[0]) {
         case 'r':

cores/esp8266/FS.h

@@ -28,6 +28,7 @@ namespace fs {
 
 class File;
 class Dir;
+class FS;
 
 class FileImpl;
 typedef std::shared_ptr<FileImpl> FileImplPtr;
@@ -48,7 +49,7 @@ enum SeekMode {
 class File : public Stream
 {
 public:
-    File(FileImplPtr p = FileImplPtr()) : _p(p) {}
+    File(FileImplPtr p = FileImplPtr(), FS *baseFS = nullptr) : _p(p), _fakeDir(nullptr), _baseFS(baseFS) { }
 
     // Print methods:
     size_t write(uint8_t) override;
@@ -72,22 +73,41 @@ class File : public Stream
     void close();
     operator bool() const;
     const char* name() const;
+    const char* fullName() const; // Includes path
+
+    bool isFile() const;
+    bool isDirectory() const;
+
+    // Arduino "class SD" methods for compatibility
+    size_t write(const char *str) { return write((const uint8_t*)str, strlen(str)); }
+    void rewindDirectory();
+    File openNextFile();
 
 protected:
     FileImplPtr _p;
+
+    // Arduino SD class emulation
+    std::shared_ptr<Dir> _fakeDir;
+    FS                  *_baseFS;
 };
 
 class Dir {
 public:
-    Dir(DirImplPtr impl = DirImplPtr()): _impl(impl) { }
+    Dir(DirImplPtr impl = DirImplPtr(), FS *baseFS = nullptr): _impl(impl), _baseFS(baseFS) { }
 
     File openFile(const char* mode);
+
     String fileName();
     size_t fileSize();
+    bool isFile() const;
+    bool isDirectory() const;
+
     bool next();
+    bool rewind();
 
 protected:
     DirImplPtr _impl;
+    FS       *_baseFS;
 };
 
 struct FSInfo {
@@ -125,6 +145,12 @@ class FS
     bool rename(const char* pathFrom, const char* pathTo);
     bool rename(const String& pathFrom, const String& pathTo);
 
+    bool mkdir(const char* path);
+    bool mkdir(const String& path);
+
+    bool rmdir(const char* path);
+    bool rmdir(const String& path);
+
 protected:
     FSImplPtr _impl;
 };

cores/esp8266/FSImpl.h

@@ -36,6 +36,9 @@ class FileImpl {
     virtual size_t size() const = 0;
     virtual void close() = 0;
     virtual const char* name() const = 0;
+    virtual const char* fullName() const = 0;
+    virtual bool isFile() const = 0;
+    virtual bool isDirectory() const = 0;
 };
 
 enum OpenMode {
@@ -57,7 +60,10 @@ class DirImpl {
     virtual FileImplPtr openFile(OpenMode openMode, AccessMode accessMode) = 0;
     virtual const char* fileName() = 0;
     virtual size_t fileSize() = 0;
+    virtual bool isFile() const = 0;
+    virtual bool isDirectory() const = 0;
     virtual bool next() = 0;
+    virtual bool rewind() = 0;
 };
 
 class FSImpl {
@@ -72,7 +78,8 @@ class FSImpl {
     virtual DirImplPtr openDir(const char* path) = 0;
     virtual bool rename(const char* pathFrom, const char* pathTo) = 0;
     virtual bool remove(const char* path) = 0;
-
+    virtual bool mkdir(const char* path) = 0;
+    virtual bool rmdir(const char* path) = 0;
 };
 
 } // namespace fs

cores/esp8266/Updater.cpp

@@ -24,7 +24,7 @@ extern "C" {
     #include "user_interface.h"
 }
 
-extern "C" uint32_t _SPIFFS_start;
+extern "C" uint32_t _FS_start;
 
 UpdaterClass::UpdaterClass()
 : _async(false)
@@ -81,8 +81,8 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
   }
 
 #ifdef DEBUG_UPDATER
-  if (command == U_SPIFFS) {
-    DEBUG_UPDATER.println(F("[begin] Update SPIFFS."));
+  if (command == U_FS) {
+    DEBUG_UPDATER.println(F("[begin] Update Filesystem."));
   }
 #endif
 
@@ -106,7 +106,7 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
     //size of current sketch rounded to a sector
     size_t currentSketchSize = (ESP.getSketchSize() + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
     //address of the end of the space available for sketch and update
-    uintptr_t updateEndAddress = (uintptr_t)&_SPIFFS_start - 0x40200000;
+    uintptr_t updateEndAddress = (uintptr_t)&_FS_start - 0x40200000;
     //size of the update rounded to a sector
     size_t roundedSize = (size + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
     //address where we will start writing the update
@@ -124,8 +124,8 @@ bool UpdaterClass::begin(size_t size, int command, int ledPin, uint8_t ledOn) {
       return false;
     }
   }
-  else if (command == U_SPIFFS) {
-     updateStartAddress = (uintptr_t)&_SPIFFS_start - 0x40200000;
+  else if (command == U_FS) {
+     updateStartAddress = (uintptr_t)&_FS_start - 0x40200000;
   }
   else {
     // unknown command
@@ -273,8 +273,8 @@ bool UpdaterClass::end(bool evenIfRemaining){
 #ifdef DEBUG_UPDATER
     DEBUG_UPDATER.printf_P(PSTR("Staged: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
   }
-  else if (_command == U_SPIFFS) {
-    DEBUG_UPDATER.printf_P(PSTR("SPIFFS: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
+  else if (_command == U_FS) {
+    DEBUG_UPDATER.printf_P(PSTR("Filesystem: address:0x%08X, size:0x%08zX\n"), _startAddress, _size);
 #endif
   }
 
@@ -386,7 +386,7 @@ bool UpdaterClass::_verifyHeader(uint8_t data) {
             return false;
         }
         return true;
-    } else if(_command == U_SPIFFS) {
+    } else if(_command == U_FS) {
         // no check of SPIFFS possible with first byte.
         return true;
     }
@@ -420,7 +420,7 @@ bool UpdaterClass::_verifyEnd() {
         }
 
         return true;
-    } else if(_command == U_SPIFFS) {
+    } else if(_command == U_FS) {
         // SPIFFS is already over written checks make no sense any more.
         return true;
     }

cores/esp8266/Updater.h

@@ -20,7 +20,7 @@
 #define UPDATE_ERROR_SIGN               (12)
 
 #define U_FLASH   0
-#define U_SPIFFS  100
+#define U_FS      100
 #define U_AUTH    200
 
 #ifdef DEBUG_ESP_UPDATER