SD library (very) slow

[kas2]

Description

SdFat library appears to be 85 time faster than Espressif SD library

I am implementing the superb ESP32-Radio with built in MP3 reader from Edzelf
https://github.com/Edzelf/ESP32-Radio

During setup(), SD card tracks are enumerated
This operation takes a very long time (several minutes for 500 MP3 tracks), which is a no go

To demonstrate the point, I prepared two minimal sketches for this SD card files count operation

SD_espressif_test_1.ino, with Espressif SD library
// Minimal sketch using Espressif SD library,
// from hardware\espressif\esp32\libraries\SD\examples\SD_test.ino

#include "FS.h"
#include "SD.h"
#include "SPI.h"

#define sd_cs_pin 21 // SD card CS pin
#define SDSPEED 20000000 // result: 101 tracks 66228ms (1'06")

long timeInit = 0;
int fileCount = 0;

void listDir(fs::FS &fs, const char * dirname, uint8_t levels) {
Serial.println("Listing directory");
File root = fs.open(dirname);
File file = root.openNextFile();
while(file) {
fileCount++;
file = root.openNextFile();
}
}

void setup() {
Serial.begin(115200);
delay(10);
delay(1000);
Serial.println("\n SD_espressif_test_1");

if(!SD.begin(sd_cs_pin, SPI, SDSPEED)) {
Serial.println("Card Mount Failed");
return;
}

timeInit = millis();
listDir(SD, "/", 0);
Serial.print(fileCount); Serial.print(" tracks\t"); // report number of tracks retrieved
Serial.print(millis() - timeInit); Serial.println("ms"); // report time to retrieve tracks
}

void loop(){} // do nothing

---

SdFAT_test_2.ino with Arduino SdFat library (https://github.com/greiman/SdFat)

// Minimal sketch using SdFat library with ESP32
// from Arduino\libraries\SdFat-master\examples\LongFileName.ino

#include<SPI.h>
#include "SdFat.h" // https://github.com/greiman/SdFat

#define sd_cs_pin 21 // SD card CS pin
#define SDSPEED SD_SCK_MHZ(20) // result: 101 tracks in 771 ms

SdFat sd;
SdFile track;

long timeInit = 0;
int fileCount = 0;

void listDir() {
while (track.openNext(sd.vwd(), O_READ)) {
fileCount++;
track.close();
}
}

void setup() {
Serial.begin(115200);
delay(10);
Serial.println("\n SdFAT_test_2");

if (!sd.begin(sd_cs_pin, SDSPEED)) {
Serial.println("Card Mount Failed");
return;
}

timeInit = millis();
listDir();
Serial.print(fileCount); Serial.print(" tracks\t"); // report number of tracks retrieved
Serial.print(millis() - timeInit); Serial.println("ms"); // report time to retrieve tracks
}

void loop() {} // do nothing

---

Using a SanDisk Ultra class10 16GB with 100 MP3 tracks in the root directory
(normally, my SD card is loaded with 3000 files ...)

Results for 100 files, @20 MHz:
•Espressif SD library: 66228 ms (1'06")
•SdFat library: 771 ms

SdFat library appears to be 85 time faster than Espressif SD library !!

Is there any means to speed up the process ??
Please let me have your comments

[everslick]

I see very poor data throughput when downloading files from SD card compared to SPIFFS.

download from SD card: <70KB/s
download from SD card: >300KB/s
download from SPIFFS: >300KB/s

i would expect the SD card be even faster or no factor at all, because network bandwidth is the bottleneck.

EDIT: I had typo that set the SPI bus speed to a too low frequency (by a factor of 10). Now file downloads are as fast as on SPIFFS.

[Shdwdrgn]

Is there any more work being done to improve the speed of SD reads? I thought it was because I was using older code so yesterday I made a clean install of Arduino IDE 1.8.5 and loaded the ESP32 library. My own testing involves opening a single file then recording the time it takes to read 4096 sequential bytes from that file. The best timing I get (when setting the SPI bus speed in the SD.begin() command) is still averaging about 16.9 microseconds per byte which comes out to around 60KB/s. It seems like @everslick 's comment above is suggesting I should be seeing more like 300KB/s or around 3.4us per read?

[everslick]

Reading single bytes will always be slow compared to block reads. Try reading 1KB instead. is it still that slow?

[Shdwdrgn]

Ah! I had no idea there was a method of reading a whole block at once. That made a huge difference in the average read time, even with smaller buffer sizes. Thanks!

[cicciocb]

Looking into vfs_api.cpp, in the function

FileImplPtr VFSFileImpl::openNextFile(const char* mode)
{
if(!_isDirectory || !_d) {
return FileImplPtr();
}
struct dirent *file = readdir(_d);
if(file == NULL) {
return FileImplPtr();
}
if(file->d_type != DT_REG && file->d_type != DT_DIR) {
return openNextFile(mode);
}
String fname = String(file->d_name);
String name = String(_path);
if(!fname.startsWith("/") && !name.endsWith("/")) {
name += "/";
}
name += fname;

return std::make_shared<VFSFileImpl>(_fs, name.c_str(), mode);  <<<-- this line

}

the line containing std::make_shared is causing the delay.
Hacking this function in order to return back just the file name become much faster, equivalent to SdFat

[GabeHC]

the line containing std::make_shared is causing the delay.
Hacking this function in order to return back just the file name become much faster, equivalent to SdFat

How? is it working?

[driftregion]

Thanks for this tip @cicciocb. The time it takes to scan a directory has dropped substantially. My patch is here: hack.patch This was applied atop arduino-esp32 commit fd5a2f0.

I'm quite curious about why std::make_shared is correlated with this behavior. I see that it allocates a shared pointer (some call it a "smart pointer").

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.

[ketan]

For anyone stumbling on this thread (since it's the first search on google for "esp32 slow sd"). See my comment here: #1117 (comment)