ESP32 Config improvement

[savejeff]

Moin Moin,

i just tested the SdFat lib on the ESP32 (using PlatformIO) and I found out it works (almost) out of the box. Great Start!

but I looked at the SdFatConfig.h and changed the SDFAT_FILE_TYPE to 3 and then it works no problem. with SDFAT_FILE_TYPE == 1 (which is selected by default because there is no check for ESP architecture) I had problems opening files like:

#define FILE_FLAG_APPEND (O_RDWR | O_CREAT | O_APPEND)
file1.open(FILENAME_FILE1, FILE_FLAG_APPEND)

I added an additional #if to check for ESP architecture resulting in:

#if defined(__AVR__) && FLASHEND < 0X8000
// 32K AVR boards.
#define SDFAT_FILE_TYPE 1
#elif defined(__arm__)
// ARM boards usually have plenty of memory
#define SDFAT_FILE_TYPE 3
#elif defined(__xtensa__)
// ESP boards usually have plenty of memory
#define SDFAT_FILE_TYPE 3
#else  // defined(__AVR__) && FLASHEND < 0X8000
// All other boards.
#define SDFAT_FILE_TYPE 1
#endif  // defined(__AVR__) && FLASHEND < 0X8000

i think __xtensa__ is also enabled for ESP8266. IMHO ESP8266 should also have enough memory.

another small thing: would It be possible to add checks if some of the configs are already defined with compiler flags? that would make it possible to use the library from github without downloading and adding it manually. that way the library stays up to date. Code like this would not change the default behavior but would it make it possible to set these flags in platformio

/**
 * If the symbol SPI_DRIVER_SELECT is:
 *
 * 0 - An optimized custom SPI driver is used if it exists
 *     else the standard library driver is used.
 *
 * 1 - The standard library driver is always used.
 *
 * 2 - An external SPI driver of SoftSpiDriver template class is always used.
 *
 * 3 - An external SPI driver derived from SdSpiBaseClass is always used.
 */
#ifndef SPI_DRIVER_SELECT
#define SPI_DRIVER_SELECT 0
#endif // SPI_DRIVER_SELECT
/**
 * If USE_SPI_ARRAY_TRANSFER is non-zero and the standard SPI library is
 * use, the array transfer function, transfer(buf, size), will be used.
 * This option will allocate up to a 512 byte temporary buffer for send.
 * This may be faster for some boards.  Do not use this with AVR boards.
 */
#ifndef USE_SPI_ARRAY_TRANSFER
#define USE_SPI_ARRAY_TRANSFER 0
#endif // USE_SPI_ARRAY_TRANSFER

[greiman]

You must not have the current version of SdFat, there are now tests to allow compiler flags. Look at SdFatConfig.h:

#ifndef SDFAT_FILE_TYPE
#if defined(__AVR__) && FLASHEND < 0X8000
// 32K AVR boards.
#define SDFAT_FILE_TYPE 1
#elif defined(__arm__)
// ARM boards usually have plenty of memory
#define SDFAT_FILE_TYPE 3
#else  // defined(__AVR__) && FLASHEND < 0X8000
// All other boards.
#define SDFAT_FILE_TYPE 1
#endif  // defined(__AVR__) && FLASHEND < 0X8000
#endif  // SDFAT_FILE_TYPE

#ifndef SPI_DRIVER_SELECT
#define SPI_DRIVER_SELECT 0
#endif  // SPI_DRIVER_SELECT

#ifndef USE_SPI_ARRAY_TRANSFER
#define USE_SPI_ARRAY_TRANSFER 0
#endif  // USE_SPI_ARRAY_TRANSFER

There are now about 200 Arduino style boards and the only boards that have too little memory are AVR 328 boards so I plan to only select SDFAT_FILE_TYPE 1 for AVR. I will test it in SdFat-beta for a while to see if there are problems.

[savejeff]

Seriously? it happened again xD. I posted a question about RaspberryPi Pico a month or so ago and it was the same case where the changes I proposed were implemented since I pulled the library version. This time I even looked over the current version of SdFatConfig.h but I only looked for a ESPor __xtensa__ clause x] Do you plan to support this library for the foreseeable future? if so I'll completely move over to SdFat for all my SD Card needs. Since about a year the SD Card Lib from ESP32 is not stable anymore. I get stuff like data is written into the wrong file if I have two file open at the same time. It only happens every hour or so and i see no chance of getting this resolved by the Espressif developers.

I just abstracted my firmware code for the sd card interaction and swapped out the ESP32 sd for the SdFat library and it seems to work flawlessly. The SdFat is even fast than the native SPI SD lib. I only swap over to the ESP Lib if i use the MMC connection.

I selected 20Mhz for the bus speed. Do you have a recommended maximum bus speed that is stable?

[greiman]

Support for SdFat is becoming difficult. I wrote the first version of SdFat in 2008 Arduino adopted a wrapper for a 2009 version and occasionally makes changes. For years conflicts were common but now Arduino seem to use more care so new versions of SdFat can coexist.

Other board packages are now becoming a major problem. Some take a version of SdFat and cause new versions of SdFat to conflict. Others wrap FatFS with an SdFat API to cause conflicts.

SdFat made sense when I wrote it for the AVR 328 Arduino as a replacement for my first library, Fat16, for the AVR 168 boards that have one KB of RAM.

Now filesystems should be integrated into the kernel together with networking. SdFat was not designed for this use. Most Arduino style boards have too little RAM to use a Linux style architecture. Modern SD card and SSDs require huge page caches for high performance.

I can't promise to keep SdFat working with any particular board in this environment since mods for one board package often breaks another package.

[savejeff]

For me personally, the SdFat fills a very important purpose. It just simply solves the task of "I have an SPI port and I want to communicate to an SD Card".

I don't know how complex the SdFat code is in the background, but if it only uses standard C code and functions, it should be easily adaptable to different boards.
are there specific includes or something like that?

a perfect version of the library would be one where I use the external SPI driver feature and it compiles everywhere. Special optimizations for specific boards/platforms are not that interesting for me. If it achieves an acceptable data rate, stability and simplicity are far more valuable (in my opinion).

integrated filesystems in arduino and microcontroller in general is kind of a nightmare for me. every vendor/arduino core has to maintain a high level of stability. if something works only 99% well and crashes 1% of the time after a 1h of writing, there is almost no chance for a developer to reach the maintainer if the core is not well maintained. for example the ESP arduino core, its very hard to raise an issue and I'm basically dependent on Espressif finding the bug on there own

[greiman]

I have tried to keep the library independent of any SPI driver. You can provide your own custom SPI driver by choosing SPI_DRIVER_SELECT == 3.

See the UserSpiDriver example. You just need to implement the SdSpiBaseClass interface.

The core libraries are designed to be used with a custom BlockDriver. I have USB examples for flash drives and even tested with an old hard drive.

I have also ported a private version of SdFat to the ChibiOS RTOS. You can find hint of these case with things like typedefs for Print and this section of SdfatConfig.h.

/** For Debug - must be one */
#define ENABLE_ARDUINO_FEATURES 1
/** For Debug - must be one */
#define ENABLE_ARDUINO_SERIAL 1
/** For Debug - must be one */
#define ENABLE_ARDUINO_STRING 1

Here is the USB flash drive test. It uses a USB host library with a mass storage driver.

// Edit SdFatConfig.h and enable generic block devices.
// #define USE_BLOCK_DEVICE_INTERFACE 1
#include "UsbMscDriver.h"

USB usb;
BulkOnly bulk(&usb);
UsbMscDriver usbKey(&bulk);

//FatVolume key;
//FatFile file;

FsVolume key;
FsFile file;

//uint8_t lun;

void setup() {
  Serial.begin(9600);
  while (!Serial) {}
  Serial.println(F("Type any character to start"));
  while (!Serial.available()) {}
 
  if (!initUSB(&usb)) {
    Serial.println("initUSB failed");
    while(1){}
  }
  // Must set USE_BLOCK_DEVICE_INTERFACE non-zero in SdFatConfig.h
  if (!key.begin(&usbKey)) {
    Serial.println("key.begin failed");
    while(1) {}
  }
  if (!file.open("usbtest.txt", FILE_WRITE)) {
    Serial.println("file.open failed");
    while(1) {}    
  }
  file.println("test line");
  file.close();
  key.ls(LS_DATE | LS_SIZE);
}

void loop() {
}

[greiman]

You have a point about micro-controller software. Companies that produce fine hardware have some of the worst software support.

I like STM32 chips but can't stand their software.

I guess most micro-controllers are embedded in mass produced products and the software is not public, probably also stuff built by an army of employees with bad design.

[savejeff]

oh yes don't get me started on STM and there HAL and 2500 chip variants. the logging framework I'm developing is cross-platform and currently works on the AVR, SAMD and EPS32. I wanted to add STM32 support but it was impossible to find example code and libraries for everything out of the typical serial, SPI and i2c stuff. I tried to get CAN to work and it immediately started with different HAL versions for different chips and "there are only minor differences between chip". I only hear bad things about STM software.

Personally, I think Arduino was a revolutionary idea. Having a cross-architecture standard for accessing basic interfaces like serial, spi and i2c should have been standard 20 years earlier. I always prefer external librarys over vendor-specific code where you are at the mercy of the developers. The ESP32 SD Card code made so many problems for me. The worst thing is that the code just halts and does not return if something goes wrong. you have no way to re-init or analyze. the firmware function just halts until the watchdog reset kicks in. There is also no way of checking if the SD card is ready to write. as a result, I can not be confident that the call is finished under 500ms. absolutely unacceptable. But as it is a company it's hard to make a feature request. The esp forum is more like an "I don't get it to work help me" support desk. The STM Core is as far as I know only half supported by STM and started as a community project. I think how the RP2040 does things is very much the modern way of doing things with easy IDE setup, easy programming, easy to access documentation and no extra hardware needed to program and debug. Meanwhile with STM you need an extra IDE, additional driver, Programmer with extra driver and debugging is also not easy to setup.

[savejeff]

About the BlockDriver: its a little bit out of the box but would it theoretically be possible to implement an MMC interface using the RP2040s PIO feature?

I'm not yet very familiar with the Storage Structure and communication with SD Cards and MMC storage devices. My guess is, if you get the MMC interface working the storage interaction would be the same as over SPI.

Its just an idea. Whats you guess how hard it would be to get this working?

[savejeff]

How did you interface the USB Flash drive? Did you directly connect the D+ D- to IO or was it a processor with full USB 2.0 support? I always though USB is to complex to use effectively on microcontroller

[greiman]

I suspect the RP2040 PIO could support MMC. There are examples of 4-bit SDIO for SD cards. Performance probably would be limited.

It would be easy to support older MMC cards on SPI but I have not added support. MMC has diverged from SD so I have hesitated since people would want to use modern cards and combining MMC and SD in one driver would add a lot of complexity with few users interested in MMC.

I have looked at eMMC and high end MMC cards. These cards either don't support SPI or have SPI support but would would be best with a different bus. The SDIO/MMC controllers on most micro-processors don't perform well since they don't allow extended transfers. Fast cards require transfers of a MB or more to achieve high speed. I can do this with SPI by putting the card in read or write mode and only interrupt the transfer when required.

I don't want to write base level MMC bus level drivers for micro-controllers, there are multiple controller versions for a single manufacturer. I already support too many custom drivers.

The NXP processor used by Teensy 3.x/4.x is an exception. I get 22 MB/sec read/write, close to the theoretical 25 MB/sec for SD cards with a 4-bit 3.3V bus. This took a lot of effort since I must have been the first to support the "infinite transfer" capability. It had bugs that required work-arounds.

The STM32H series has a SDIO controller that looks promising but the series still has too many errata.

For MMC, I would implement a separate driver if I supported it.

I tested the USB cards and disks using a USB host shield and this library.

SdFat has been used on Teensy with the NXP USB controller in host mode. It is likely that any implementation of the USB mass storage protocol could be wrapped with my BlockDevice Interface.