SD library

[ladyada]

did some minor tweaks to ESP8266 lib and got it workin'

adafruit@15bace7

its not terribly fast but does work. lemme know if you want a pull req

[me-no-dev]

As a matter of fact I am currently looking into the SD library :) More so on the actual SD/MMC bus that is on the ESP32 and that will maybe drive the SD library. Also I have running a slight modification to the mbed SD and FAT FS code. To be honest I think that we should replace the whole SD lib with something better and more suitable for the faster and more capable ESP32 chip that this AVR 8 bit compatible lib :)

[ladyada]

the SD library could be re-written but it does the heavy lifting of decoding a FAT16/32 file system which is the most annoying part of an SD library. the reason the performance is sluggish is probably due to SPI.transfer() taking 30us even if the clock speed is 32mhz. e.g. if you call SPI.transfer() twice in a row it puts a huge delay essnetially turning the SPI from 32MHz to 260KHz clk!

on most chipsets the transfer itself quite fast, but each transfer on the ESP32 requires a mutex that's probably where the massive delay is coming from. so probably the best bet is just to rewrite the low level parts of SD to not call SPI.transfer multiple times, and use the buffered version - with SD you could probably blit all 512 bytes of a block in the same time of 4 non-buffered reads! probably can spare 2 block buffers on the ESP32 anyhow

i can take a look at that and just give you one PR if you want

[me-no-dev]

I would like to give mbed's fat and sd libs a go first as they are separate things (FATFS is one thing, SD control another thing) which makes FATFS implementable on any storage and not just SD. I guess that hard lifting you are talking about. I will optimize the SPI for it as much as it can be done, but SPI will not be the only way to access SD cards. As mentioned there is SD/MMC hardware bus on the ESP32, which reads the card in 4 bit mode (just like the SPI flash we use for the sketch) and at the end you will be able to use either interface, or even two cards at the same time.

[me-no-dev]

To add to that, I turned mbed's fatfs to C library which looks muck like the SPIFFS C api and we will need to implement both in esp-idf as well. Arduino will actually attach to IDF and expose the libs through there.

[ladyada]

sounds fine by me! i'll continue using SD for now

[Curclamas]

@me-no-dev Just because I am also right now working on a SD-related project:
What is your status on the SPI-SD-implementation? Will it be part of the arduino-esp32 project will der be official SD via SPI support with FAT32 in ESP-IDF?
I'd be quite happy if you could give me some feedback on that.

[me-no-dev]

as a matter of fact sdmmc driver and fatfs are waiting for merge :) so expect a fast and reliable native SD/MMC support quite soon.

[Curclamas]

Thanks for the quick reply! You are talking about IDF here right? So does this mean only "MMC" mode acces is possible or will there also be the possibility for SPI access? Is there a time frame for the merge?
Also if SD/MMC will be implemented in IDF will there be a "Port" for the Arduino SD class?

[igrr]

Actually MMC will not initially be supported. At first we will only support SD cards in ESP-IDF (via SD protocol, not SPI). Support for SPI mode will also be added, but maybe in the next release. Then we can also support MMC/eMMC cards via MMC protocol, if there is request for that.

Arduino libraries may be based upon ESP-IDF, if desired. It's mostly up to @me-no-dev.

[me-no-dev]

Arduino libs for SD/SPIFFS and similar will surely be based on IDF libs. Hardware drivers are usually the only exclusion, because Arduino has different approach towards some of them.
I would like to start from new FileSystem classes that are compatible with Arduino's SD and out ESP8266 SPIFFS, in hope to smooth out the issues we have been having when using both, or implementing libs that want to use both

[Dennis650]

Hi,

i just downloaded the libraries/SD folder (adafruit@15bace7) to my Arduino libraries. But it didn't compile.

What do i have to do?

Greetings
Dennis

20:04:00 **** Incremental Build of configuration Release for project CamperControl ****
make all 
Building file: ../.ino.cpp
Starting C++ compile
"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++" -DESP_PLATFORM -DMBEDTLS_CONFIG_FILE='"mbedtls/esp_config.h"' -DHAVE_CONFIG_H "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/config" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/bt" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/driver" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/esp32" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/ethernet" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/freertos" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/log" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/vfs" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/ulp" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/nvs_flash" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/spi_flash" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/openssl" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/app_update" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/tcpip_adapter" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/xtensa-debug-module" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/wpa_supplicant" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/expat" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/json" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/mbedtls" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/nghttp" "-I/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/lwip" -std=gnu++11 -fno-exceptions -fno-rtti -Os -g3 -Wpointer-arith -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -mlongcalls -nostdlib -Wno-error=unused-function -Wno-error=unused-but-set-variable -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-sign-compare -fno-rtti -MMD -c -DF_CPU=240000000L -DARDUINO=10609 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 -DESP32  -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/cores/esp32" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/variants/esp32" -I"/Users/dennis/Documents/Arduino/libraries/DS2438" -I"/Users/dennis/Documents/Arduino/libraries/OneWire" -I"/Users/dennis/Documents/Arduino/libraries/Ucglib" -I"/Users/dennis/Documents/Arduino/libraries/Ucglib/src" -I"/Users/dennis/Documents/Arduino/libraries/Arduino-Temperature-Control-Library-master" -I"/Users/dennis/Documents/Arduino/libraries/arduino-2406-master" -I"/Users/dennis/Documents/Arduino/libraries/Arduino-DCF77-master" -I"/Users/dennis/Documents/Arduino/libraries/Arduino-DCF77-master/utility" -I"/Users/dennis/Documents/Arduino/libraries/Timezone-master" -I"/Users/dennis/Documents/Arduino/libraries/Time-master" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/WiFi" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/WiFi/src" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SPI" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SPI/src" -I"/Users/dennis/Documents/Arduino/libraries/XPT2046_Touchscreen-master" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD" -I"/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src" -MMD -MP -MF".ino.cpp.d" -MT".ino.cpp.o" -D__IN_ECLIPSE__=1 -x c++ "../.ino.cpp" -o ".ino.cpp.o"  -Wall
In file included from ../.ino.cpp:8:0:
/Users/dennis/Documents/Arduino/libraries/OneWire/OneWire.h:249:2: warning: #warning "OneWire. Fallback mode. Using API calls for pinMode,digitalRead and digitalWrite. Operation of this library is not guaranteed on this architecture." [-Wcpp]
 #warning "OneWire. Fallback mode. Using API calls for pinMode,digitalRead and digitalWrite. Operation of this library is not guaranteed on this architecture."
  ^
In file included from ../.ino.cpp:11:0:
/Users/dennis/Documents/Arduino/libraries/DS2438/DS2438.h:35:0: warning: "TEMP_LSB" redefined [enabled by default]
 #define TEMP_LSB  1
 ^
In file included from ../.ino.cpp:10:0:
/Users/dennis/Documents/Arduino/libraries/Arduino-Temperature-Control-Library-master/DallasTemperature.h:40:0: note: this is the location of the previous definition
 #define TEMP_LSB        0
 ^
In file included from ../.ino.cpp:11:0:
/Users/dennis/Documents/Arduino/libraries/DS2438/DS2438.h:36:0: warning: "TEMP_MSB" redefined [enabled by default]
 #define TEMP_MSB  2
 ^
In file included from ../.ino.cpp:10:0:
/Users/dennis/Documents/Arduino/libraries/Arduino-Temperature-Control-Library-master/DallasTemperature.h:41:0: note: this is the location of the previous definition
 #define TEMP_MSB        1
 ^
In file included from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/fcntl.h:3:0,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/lwip/lwipopts.h:38,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/lwip/lwip/opt.h:45,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/lwip/lwip/ip_addr.h:35,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/tcpip_adapter/tcpip_adapter.h:47,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/esp32/esp_event.h:23,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/esp32/esp_event_loop.h:22,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/WiFi/src/WiFiGeneric.h:28,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/WiFi/src/WiFiSTA.h:28,
                 from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/WiFi/src/WiFi.h:32,
                 from ../.ino.cpp:14:
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:34:18: error: expected unqualified-id before numeric constant
 #define O_RDONLY 0  /* +1 == FREAD */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:55:15: note: in expansion of macro 'O_RDONLY'
 uint8_t const O_RDONLY = O_READ;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:35:18: error: expected unqualified-id before numeric constant
 #define O_WRONLY 1  /* +1 == FWRITE */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:59:15: note: in expansion of macro 'O_WRONLY'
 uint8_t const O_WRONLY = O_WRITE;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:36:17: error: expected unqualified-id before numeric constant
 #define O_RDWR  2  /* +1 == FREAD|FWRITE */
                 ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:61:15: note: in expansion of macro 'O_RDWR'
 uint8_t const O_RDWR = (O_READ | O_WRITE);
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:34:18: error: expected unqualified-id before numeric constant
 #define O_RDONLY 0  /* +1 == FREAD */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:27:20: note: in expansion of macro 'O_RDONLY'
 #define O_ACCMODE (O_RDONLY|O_WRONLY|O_RDWR)
                    ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:63:15: note: in expansion of macro 'O_ACCMODE'
 uint8_t const O_ACCMODE = (O_READ | O_WRITE);
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:34:18: error: expected ')' before numeric constant
 #define O_RDONLY 0  /* +1 == FREAD */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:27:20: note: in expansion of macro 'O_RDONLY'
 #define O_ACCMODE (O_RDONLY|O_WRONLY|O_RDWR)
                    ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:63:15: note: in expansion of macro 'O_ACCMODE'
 uint8_t const O_ACCMODE = (O_READ | O_WRITE);
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:12:18: error: expected unqualified-id before numeric constant
 #define _FAPPEND 0x0008 /* append (writes guaranteed at the end) */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:37:18: note: in expansion of macro '_FAPPEND'
 #define O_APPEND _FAPPEND
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:65:15: note: in expansion of macro 'O_APPEND'
 uint8_t const O_APPEND = 0X04;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:22:17: error: expected unqualified-id before numeric constant
 #define _FSYNC  0x2000 /* do all writes synchronously */
                 ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:41:17: note: in expansion of macro '_FSYNC'
 #define O_SYNC  _FSYNC
                 ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:67:15: note: in expansion of macro 'O_SYNC'
 uint8_t const O_SYNC = 0X08;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:18:18: error: expected unqualified-id before numeric constant
 #define _FCREAT  0x0200 /* open with file create */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:38:18: note: in expansion of macro '_FCREAT'
 #define O_CREAT  _FCREAT
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:69:15: note: in expansion of macro 'O_CREAT'
 uint8_t const O_CREAT = 0X10;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:20:17: error: expected unqualified-id before numeric constant
 #define _FEXCL  0x0800 /* error on open if file exists */
                 ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:40:17: note: in expansion of macro '_FEXCL'
 #define O_EXCL  _FEXCL
                 ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:71:15: note: in expansion of macro 'O_EXCL'
 uint8_t const O_EXCL = 0X20;
               ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:19:18: error: expected unqualified-id before numeric constant
 #define _FTRUNC  0x0400 /* open with truncation */
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/tools/sdk/include/newlib/sys/_default_fcntl.h:39:18: note: in expansion of macro '_FTRUNC'
 #define O_TRUNC  _FTRUNC
                  ^
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:73:15: note: in expansion of macro 'O_TRUNC'
 uint8_t const O_TRUNC = 0X40;
               ^
In file included from /Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/SD.h:20:0,
                 from ../.ino.cpp:17:
/Users/dennis/Documents/Arduino/hardware/espressif/esp32/libraries/SD/src/utility/SdFat.h:374:64: warning: large integer implicitly truncated to unsigned type [-Woverflow]
   static uint8_t const F_OFLAG = (O_ACCMODE | O_APPEND | O_SYNC);
                                                                ^
make: *** [.ino.cpp.o] Error 1

20:04:00 Build Finished (took 436ms)

[marcmerlin]

So, I'd like to add sdcard support to an ESP32 project, using the arduino libs.
What is the current working solution (ok if not fast), if any?
Thanks.

[me-no-dev]

please hold on! I'm waiting on some changes to be merged upstream (esp-idf) to release the new file system libs (including SD and SD_MMC). Everything is ready so will be up soon.

[marcmerlin]

Hi @me-no-dev , is this still blocked with the PR you're trying to get merged in?

[me-no-dev]

Yes. I was told it will be merged after idf2.0 release

[marcmerlin]

Sorry for not knowing the timeline, is that on the order of days, weeks, months ? Thanks

…

On Feb 18, 2017 06:54, "Me No Dev" ***@***.***> wrote: Yes. I was told it will be merged after idf2.0 release — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABTlRBLh-lIeWxIVKUtb1xT4Ow-3U4e9ks5rdwYvgaJpZM4KwVy7> .

[me-no-dev]

days, weeks

[krazatchu]

Thank you!

[me-no-dev]

:)

[LongmontRobotics]

Has this been completed and merged ?

[me-no-dev]

yes. long time ago

[andig]

@me-no-dev does that mean that we also have spiffs support now (or something that looks like spiffs)?

[me-no-dev]

@andig SPIFFS is about to be merged. There is SD and SD_MMC to work with cards. All 3 are based on VFS so API is the same.

[LongmontRobotics]

Thank You

[namork]

dear all

i'm struggling to get the sd card working on a lolin32 pro. it looks like the pins are wired differently. where can i change the pins in the library or in the code itself?

thanks a lot!