Add a FAT filesystem for SD cards to Arduino FS

Arduino forked a copy of SD lib several years ago, put their own wrapper
around it, and it's been languishing in our ESP8266 libraries ever since
as SD. It doesn't support long file names, has class names which
conflict with the ESP8266 internal names, and hasn't been updated in
ages.

The original author of the SD library has continued work in the
meantime, and produced a very feature rich implementation of SdFat. It
unfortunately also conflicts with the class names we use in ESP8266
Arduino and has a different API than the internal SPIFFS or proposed
LittleFS filesystem objects.

This PR puts a wrapper around the latest and greatest SdFat library,
by forking it and wrapping its classes in a private namespace "sdfat,"
and making as thin a wrapper as possible around it to conform to
the ESP8266 FS, File, and Dir classes.

This PR also removes the Arduino SD.h class library and rewrites it
using the new SDFS filesystem to make everything in the ESP8266
Arduino core compatible with each other.

By doing so it lets us use a single interface for anything needing a
file instead of multiple ones (see SDWebServer and how a different
object is needed vs. one serving from SPIFFS even though the logic is
all the same). Same for BearSSL's CertStores and probably a few others
I've missed, cleaning up our code base significantly.

Like LittleFS, silently create directories when a file is created with
a subdirectory specifier ("/path/to/file.txt") if they do not yet exist.

Adds a blacklist of sketches to skip in the CI process (because SdFat
has many examples which do not build properly on the ESP8266).

Now that LittleFS and SDFS have directory support, the FS needs to be
able to communicate whether a name is one or the other.  Add a simple
bool FS::isDirectory() and bool FS::isFile() method.  SPIFFS doesn't
have directories, so if it's valid it's a file and reported as such.

Add ::mkdir/::rmdir to the FS class to allow users to make and destroy
subdirectories.  SPIFFS directory operations will, of course, fail
and return false.

Emulate a 16MB SD card and allow test runner to exercise it by using
a custom SdFat HOST_MOCK-enabled object.

Throw out the original Arduino SD.h class and rewrite from scratch using
only the ESP8266 native SDFS calls.  This makes "SD" based applications
compatible with normal ESP8266 "File" and "FS" and "SPIFFS" operations.

The only major visible change for users is that long filenames now are
fully supported and work without any code changes.  If there are static
arrays of 11 bytes for old 8.3 names in code, they will need to be
adjusted.

While it is recommended to use the more powerful SDFS class to access SD
cards, this SD.h wrapper allows for use of existing Arduino libraries
which are built to only with with that SD class.

Additional helper functions added to ESP8266 native Filesystem:: classes
to help support this portability.

The rewrite is good enough to run the original SDWebServer and SD
example code without any changes.

Author: earlephilhower

.gitmodules

@@ -7,3 +7,6 @@
 [submodule "libraries/SoftwareSerial"]
 	path = libraries/SoftwareSerial
 	url = https://github.com/plerup/espsoftwareserial.git
+[submodule "libraries/ESP8266SdFat"]
+	path = libraries/ESP8266SdFat
+	url = https://github.com/earlephilhower/ESP8266SdFat.git

cores/esp8266/FS.cpp

@@ -121,6 +121,44 @@ const char* File::name() const {
     return _p->name();
 }
 
+const char* File::fullName() const {
+    if (!_p)
+        return nullptr;
+
+    return _p->fullName();
+}
+
+bool File::isFile() const {
+    if (!_p)
+        return false;
+
+    return _p->isFile();
+}
+
+bool File::isDirectory() const {
+    if (!_p)
+        return false;
+
+    return _p->isDirectory();
+}
+
+void File::rewindDirectory() {
+    if (!_fakeDir) {
+        _fakeDir = std::make_shared<Dir>(_baseFS->openDir(fullName()));
+    } else {
+        _fakeDir->rewind();
+   }
+}
+
+File File::openNextFile() {
+    if (!_fakeDir) {
+        _fakeDir = std::make_shared<Dir>(_baseFS->openDir(fullName()));
+    }
+    _fakeDir->next();
+    return _fakeDir->openFile("r");
+}
+
+
 File Dir::openFile(const char* mode) {
     if (!_impl) {
         return File();
@@ -133,7 +171,7 @@ File Dir::openFile(const char* mode) {
         return File();
     }
 
-    return File(_impl->openFile(om, am));
+    return File(_impl->openFile(om, am), _baseFS);
 }
 
 String Dir::fileName() {
@@ -152,6 +190,20 @@ size_t Dir::fileSize() {
     return _impl->fileSize();
 }
 
+bool Dir::isFile() const {
+    if (!_impl)
+        return false;
+
+    return _impl->isFile();
+}
+
+bool Dir::isDirectory() const {
+    if (!_impl)
+        return false;
+
+    return _impl->isDirectory();
+}
+
 bool Dir::next() {
     if (!_impl) {
         return false;
@@ -160,6 +212,14 @@ bool Dir::next() {
     return _impl->next();
 }
 
+bool Dir::rewind() {
+    if (!_impl) {
+        return false;
+    }
+
+    return _impl->rewind();
+}
+
 bool FS::begin() {
     if (!_impl) {
         return false;
@@ -202,8 +262,7 @@ File FS::open(const char* path, const char* mode) {
         DEBUGV("FS::open: invalid mode `%s`\r\n", mode);
         return File();
     }
-
-    return File(_impl->open(path, om, am));
+    return File(_impl->open(path, om, am), this);
 }
 
 bool FS::exists(const char* path) {
@@ -221,7 +280,8 @@ Dir FS::openDir(const char* path) {
     if (!_impl) {
         return Dir();
     }
-    return Dir(_impl->openDir(path));
+    DirImplPtr p = _impl->openDir(path);
+    return Dir(p, this);
 }
 
 Dir FS::openDir(const String& path) {
@@ -239,6 +299,28 @@ bool FS::remove(const String& path) {
     return remove(path.c_str());
 }
 
+bool FS::rmdir(const char* path) {
+    if (!_impl) {
+        return false;
+    }
+    return _impl->rmdir(path);
+}
+
+bool FS::rmdir(const String& path) {
+    return rmdir(path.c_str());
+}
+
+bool FS::mkdir(const char* path) {
+    if (!_impl) {
+        return false;
+    }
+    return _impl->mkdir(path);
+}
+
+bool FS::mkdir(const String& path) {
+    return mkdir(path.c_str());
+}
+
 bool FS::rename(const char* pathFrom, const char* pathTo) {
     if (!_impl) {
         return false;
@@ -251,6 +333,7 @@ bool FS::rename(const String& pathFrom, const String& pathTo) {
 }
 
 
+
 static bool sflags(const char* mode, OpenMode& om, AccessMode& am) {
     switch (mode[0]) {
         case 'r':

cores/esp8266/FS.h

@@ -28,6 +28,7 @@ namespace fs {
 
 class File;
 class Dir;
+class FS;
 
 class FileImpl;
 typedef std::shared_ptr<FileImpl> FileImplPtr;
@@ -48,7 +49,7 @@ enum SeekMode {
 class File : public Stream
 {
 public:
-    File(FileImplPtr p = FileImplPtr()) : _p(p) {}
+    File(FileImplPtr p = FileImplPtr(), FS *baseFS = nullptr) : _p(p), _fakeDir(nullptr), _baseFS(baseFS) { }
 
     // Print methods:
     size_t write(uint8_t) override;
@@ -72,22 +73,41 @@ class File : public Stream
     void close();
     operator bool() const;
     const char* name() const;
+    const char* fullName() const; // Includes path
+
+    bool isFile() const;
+    bool isDirectory() const;
+
+    // Arduino "class SD" methods for compatibility
+    size_t write(const char *str) { return write((const uint8_t*)str, strlen(str)); }
+    void rewindDirectory();
+    File openNextFile();
 
 protected:
     FileImplPtr _p;
+
+    // Arduino SD class emulation
+    std::shared_ptr<Dir> _fakeDir;
+    FS                  *_baseFS;
 };
 
 class Dir {
 public:
-    Dir(DirImplPtr impl = DirImplPtr()): _impl(impl) { }
+    Dir(DirImplPtr impl = DirImplPtr(), FS *baseFS = nullptr): _impl(impl), _baseFS(baseFS) { }
 
     File openFile(const char* mode);
+
     String fileName();
     size_t fileSize();
+    bool isFile() const;
+    bool isDirectory() const;
+
     bool next();
+    bool rewind();
 
 protected:
     DirImplPtr _impl;
+    FS       *_baseFS;
 };
 
 struct FSInfo {
@@ -125,6 +145,12 @@ class FS
     bool rename(const char* pathFrom, const char* pathTo);
     bool rename(const String& pathFrom, const String& pathTo);
 
+    bool mkdir(const char* path);
+    bool mkdir(const String& path);
+
+    bool rmdir(const char* path);
+    bool rmdir(const String& path);
+
 protected:
     FSImplPtr _impl;
 };

cores/esp8266/FSImpl.h

@@ -36,6 +36,9 @@ class FileImpl {
     virtual size_t size() const = 0;
     virtual void close() = 0;
     virtual const char* name() const = 0;
+    virtual const char* fullName() const = 0;
+    virtual bool isFile() const = 0;
+    virtual bool isDirectory() const = 0;
 };
 
 enum OpenMode {
@@ -57,7 +60,10 @@ class DirImpl {
     virtual FileImplPtr openFile(OpenMode openMode, AccessMode accessMode) = 0;
     virtual const char* fileName() = 0;
     virtual size_t fileSize() = 0;
+    virtual bool isFile() const = 0;
+    virtual bool isDirectory() const = 0;
     virtual bool next() = 0;
+    virtual bool rewind() = 0;
 };
 
 class FSImpl {
@@ -72,7 +78,8 @@ class FSImpl {
     virtual DirImplPtr openDir(const char* path) = 0;
     virtual bool rename(const char* pathFrom, const char* pathTo) = 0;
     virtual bool remove(const char* path) = 0;
-
+    virtual bool mkdir(const char* path) = 0;
+    virtual bool rmdir(const char* path) = 0;
 };
 
 } // namespace fs

cores/esp8266/spiffs_api.h

@@ -124,6 +124,18 @@ class SPIFFSImpl : public FSImpl
         return true;
     }
 
+    bool mkdir(const char* path) override
+    {
+        (void)path;
+        return false;
+    }
+
+    bool rmdir(const char* path) override
+    {
+        (void)path;
+        return false;
+    }
+
     bool begin() override
     {
 #if defined(ARDUINO) && !defined(CORE_MOCK)
@@ -376,6 +388,18 @@ class SPIFFSFileImpl : public FileImpl
         return _stat.size;
     }
 
+    bool isFile() const override
+    {
+        // No such thing as directories on SPIFFS
+        return _fd ? true : false;
+    }
+
+    bool isDirectory() const override
+    {
+        // No such thing as directories on SPIFFS
+        return false;
+    }
+
     void close() override
     {
         CHECKFD();
@@ -391,6 +415,11 @@ class SPIFFSFileImpl : public FileImpl
         return (const char*) _stat.name;
     }
 
+    const char* fullName() const override
+    {
+        return name(); // No dirs, they're the same on SPIFFS
+    }
+
 protected:
     void _getStat() const
     {
@@ -416,6 +445,7 @@ class SPIFFSDirImpl : public DirImpl
         : _pattern(pattern)
         , _fs(fs)
         , _dir(dir)
+        , _dirHead(dir)
         , _valid(false)
     {
     }
@@ -459,6 +489,18 @@ class SPIFFSDirImpl : public DirImpl
         return _dirent.size;
     }
 
+    bool isFile() const override
+    {
+        // No such thing as directories on SPIFFS
+        return _valid;
+    }
+
+    bool isDirectory() const override
+    {
+        // No such thing as directories on SPIFFS
+        return false;
+    }
+
     bool next() override
     {
         const int n = _pattern.length();
@@ -469,10 +511,18 @@ class SPIFFSDirImpl : public DirImpl
         return _valid;
     }
 
+    bool rewind() override
+    {
+        _dir = _dirHead;
+        _valid = false;
+        return true;
+    }
+
 protected:
     String _pattern;
     SPIFFSImpl* _fs;
     spiffs_DIR  _dir;
+    spiffs_DIR  _dirHead; // The pointer to the start of this dir
     spiffs_dirent _dirent;
     bool _valid;
 };

libraries/ESP8266SdFat

@@ -0,0 +1,9 @@
+# Arduino "class SD" shim wrapper
+
+This is a simple wrapper class to replace the ancient Arduino SD.h
+access method for SD cards.  It calls the underlying SDFS and the latest
+SdFat lib to do all the work, and is now compatible with the rest of the
+ESP8266 filesystem things.
+
+-Earle F. Philhower, III
+ <[email protected]>