Experience in using SDFat for ESP32 with esp-idf

[jeromeDms]

Hi All According to the ESP32 forum, this SdFat library seems faster than the implementation of FATFs in esp-idf libraries. I'm facing some difficulties keeping a continuous write to the SD Card when recording a wave file @ 24bits 96KHz Stereo, sometimes the SD card is "busy" and writing a bloc of 16K requires a long time.

I wanted to test this library in my project, I'm not using Arduino, I'm developing with the standard esp-idf libraries. Any experience on using this library with ESP32-IDF ? Thanks for any input.

[greiman]

The problem with fast writes to SD cards is that modern SD cards emulate 512 byte sectors in a much larger RU, Recording Unit. RUs are part of a larger AU, Allocation Unit.

As you write, the card prepares new erased AUs by copying use RUs from the old AU and writing new RUs. The card may indicate busy during the time the card controller is writing and moving data.

For a class 10 SDHC card, AU size is up to 4MB and RU size is 512 KB.

You must design your app to buffer data during the busy periods. This requires doing raw writes to a large contiguous block on the card.

See the SdFat LowLatencyLogger example.

Here is the key section at about line 450.

   if (fullHead == fullTail) {
      // Exit loop if done.
      if (closeFile) {
        break;
      }
    } else if (!sd.card()->isBusy()) {
      // Get address of block to write.
      block_t* pBlock = fullQueue[fullTail];
      fullTail = fullTail < QUEUE_LAST ? fullTail + 1 : 0;
      // Write block to SD.
      uint32_t usec = micros();
      if (!sd.card()->writeData((uint8_t*)pBlock)) {
        error("write data failed");
      }
      usec = micros() - usec;
      if (usec > maxLatency) {
        maxLatency = usec;
      }
      // Move block to empty stack.
      emptyStack[emptyTop++] = pBlock;
      bn++;
      if (bn == FILE_BLOCK_COUNT) {
        // File full so stop
        break;
      }
    }

On an Arduino Due, the max time to write a 512 byte sector is 106 µs. I have not tested on ESP32.

You will need sufficient buffers for your data rate. You need a dedicated SPI port for the SD card.

For more info see section 4.13.1 of the SD Simplified Specification.

[jeromeDms]

Thanks for your answer. I've already allocated 256 buffers of 16K (4MB) ! And yes I'm using the SD with dedicated port on the ESP32. Btw, the SD is working in 4 bits mode, I guess it should be faster than SPI one bit. What I observe is some busy periods of more than 1sec, while continuously writing 16KB blocs. Is this something you already observed or am I doing something wrong ? Again, thanks for your time.

[greiman]

You are not doing raw writes like the LowLatencyLogger example and observing SD busy.

The LowLatencyLogger does one huge multi-sector raw write of the entire file, up to the max 4GB file size.

This statement transfers one sector of a unlimited length multi-sector write.

sd.card()->writeData((uint8_t*)pBlock)

Your file level 16K writes causes an access to the two copies of the FAT for every 32K cluster. This is one or more extra reads and two extra writes to allocate a new 32K cluster. It totally kills the internal buffering in the SD.

Using this method on a Due I get 4.5 MB/sec and the max time to transfer a 512 byte sector is always under 150 µs and averages about 106 µs. This is using SPI at 48 MHz.

I can log 16-bit ADC data on a STM32 at 1,000,000 samples per second without dropping samples.

Did you read section 4.13.1 of the SD Physical Layer Simplified Specification to understand the problem?

[jeromeDms]

Thanks for the detailed explanation. I read the section you mentioned, so as far as I understand, the AU is 32k by default on FAT32 as cards, and I should write buffers that are multiple of this AU size to avoid fragmented AUs. So writing blocs of 32k, 64k, 96k, ... should help reducing busy periods. Am I understanding correctly ? Thanks

[jeromeDms]

I found my issue I think. On the ESP32 I was using large circular buffers, stored in external SPI RAM, sending those buffers to fwrite() were causing problems, seems like the SPI RAM access was very slow. I checked using a single buffer located into the ESP32 internal RAM, and I did not longer observe SDcard hanging for several hundreds of milliseconds. Will further investigate on this, but 24/96/stereo seems to work fine now.

[greiman]

The file system's 32 KB cluster has nothing to do with the delay. It is the physical layout of flash in the SD.

Flash pages are huge in the SD, 512 KB in a modern SD. To achieve really high speeds you must account for this size.

You probably can get away with your buffering scheme since 24/96/stereo is fairly slow. I assume the the rate for a wave file is 2X3X96 KB/sec which is under 600 KB/sec.

Writing blocks larger than 32 KB won't help because SdFat limits internal writes to the current cluster. Unless you write on cluster boundaries your 32 KB buffer will result in two writes since the writes will not be aligned on cluster boundaries.

If you make a true wave file, writing the header can cause miss alignment so your writes are not aligned with clusters or sectors. if possible try to at least maintain writes on sector boundaries.

Probably none of this matters since you have so much RAM and a fairly slow data rate.

I didn't calculate the rate and was thinking of fast logging like three 2.4 msps ADCs on a STM32 which can deliver over 10 MB/sec.

[jeromeDms]

hmmm nor sure to clearly understand everything. Yes the recording rate is about 600KB/sec So the most efficient write size would be 32KB ?

The wave header is included in the first buffer, so everything is aligned. As I mentioned in my previous post, using external SPI flash does not work, so I have to use internal RAM which is limited.

[greiman]

If all writes are aligned on 32KB boundaries 32KB will be optimal. SdFat will do a single 32 KB write to the SD.

SdFat writes a max of one 32KB cluster in a loop then either allocates another cluster or finds the next cluster when rewriting a file.

I have been working on a new SD library that has better support for pre-allocated contiguous files. It also supports exFat. exFat allows huge 32MB clusters and standard pre-allocated contiguous files.

The SdFs library was a first cut and has better performance than the current SdFat. I intend to use SdFs as the basis for Version 2 of SdFat.

[jeromeDms]

thanks !

The only remaining problem is that your library cannot be used in ESP-IDF for ESP32 :-)

[greiman]

Looks like ESP-IDF has a SDIO driver that would work with SdFat V2.

Sadly it is typical of micro-controller drivers and lacks features for really high speed.

I did a driver for Teensy that does over 20 MB/sec with buffers as small as 512 bytes. Max possible speed is 25 MB/sec for the 50 MHz bus on Teensy 3.6

The new STM32H7 chips have a 200 MB/sec 8-bit MMC bus. I have several boards and am experimenting. I will just use the 108 MB/sec 4-bit SDIO bus.

[ModistESP32]

Hello there, I'm interested in logging data directly in Raw bytes on my non partioned sd Card. I know the commands of "writeBlock", writeStart, writeData, writeStop to do this. Unfortunatly I'm only able to get a writing speed of around 400KB/s with my ESP32 and a class 10 16gb microsdhc.

You wrote:

The problem with fast writes to SD cards is that modern SD cards emulate 512 byte sectors in a much larger RU, Recording Unit. RUs are part of a larger AU, Allocation Unit. As you write, the card prepares new erased AUs by copying use RUs from the old AU and writing new RUs. The card may indicate busy during the time the card controller is writing and moving data. For a class 10 SDHC card, AU size is up to 4MB and RU size is 512 KB. You must design your app to buffer data during the busy periods. This requires doing raw writes to a large contiguous block on the card.

That means to me that you are using raw writes on the SD to buffer data. So writing raw on the SD does not get the "sd.busy" problem when writing contiguous with writeStart->writeData->writeStop? If that's the case, why do I only achieve 400KB/s while you are able to get 4,5MB/s with Uno.

Unfortunatly i wasn't able to test the LowLatencyLogger example yet because I always get overrun error immediatly after starting the "record data" function.

Hope you got the time to answer. best regards.

[greiman]

ModistESP32: I get high speeds to files. With a Teensy 3.6 board I can write a file at about 20 MB/sec.

Teensy 3.6 has a built-in microSD socket and a 4-bit wide SDIO controller that runs at near 50 MHz.

The key to high speed is a fast SPI or SDIO controller and using SdFat in a mode that optimizes writes.

The standard ESP32 SPI controller is slow so it can't do more than about 800 KB/sec and that requires dedicated use of the SPI port.

Here are results for my ESP32 development board with 25 MHz SPI clock. Even though the clock is at 25 MHz there are large gaps between bytes so you don't get 3 MB/sec.

Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
825.57,639,613,619
825.30,639,613,619

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
826.80,622,613,618
826.80,623,613,618

This test was done with the SdFat-beta bench example. I modified this line in the example to run at 25 MHz.

#elif ENABLE_DEDICATED_SPI
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(25))
#else  // HAS_SDIO_CLASS

Edit: Here are Teensy 3.6 results for the bench example and the same SD:

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
19998.72,4039,24,25
20160.00,1217,24,24

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
20406.86,918,24,24
20406.86,127,24,24

[ModistESP32]

Hey greiman, thanks a lot for your response. That is really helpfull. I achieved the 400kb/s with the sd2card library because I thought that sdfat is based on it. Now i used Sdfat with FreeStack.h and only did raw writes and I get the speed that you got with your ESP32 board so around 820KB/s. It is really interesting that the Teensy is so much faster.

So thanks again :). I hope 800KB/s is fast enough for my purposes.

[greiman]

The Teensy 3.6 is 25 times faster than the ESP32 because of the excellent ARM Cortex M4 CPU, NPX I/O system, and systems software.

ESP32 is deficient in all these areas.

I am no longer advising use of raw writes since SdFat V2 has better support for SD busy. This example achieves the same performance as raw writes with file I/O.

[xdevelnet]

Hi, @greiman !

Thanks for such great library! I would like to ask you few questions:

1. Are you planning to make your project library-independent? The thing is, that you are using Arduino library, while I'm using esp-idf, therefore, arduino is not available
2. Do you have any plans to rewrite this library to C? I'm not a big fan of C++ and Arduino...
3. If you're not planning to rewrite it, how do you think it would be complicated? What C++ features you are using except namespaces? Do you have any suggestions for porting it to C?

I'm just looking for any library that supports exfat, but bundled exfat with esp-idf kinda bad. I mean, I can't even force the library to use this filesystem... Sigh... If you have any suggestions - I would be so happy!

Have a nice day!

[greiman]

Are you planning to make your project library-independent

The base classes are independent of Arduino and I have used them on the ChibiOS RTOS. CHIBIOS is C based.

Do you have any plans to rewrite this library to C? I'm not a big fan of C++ and Arduino...

No, I am not a big fan of C. I liked C in the 1970s when UNIX was on PDP-11.

If you're not planning to rewrite it, how do you think it would be complicated? What C++ features you are using except namespaces? Do you have any suggestions for porting it to C?

I use templates, classes in ways I can't imagine moving to C.

[greiman]

Here is a list of classes by file and line:

Search "^class" (67 hits in 39 files of 114 searched) C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\BufferedPrint.h (1 hit) Line 40: class BufferedPrint { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\common\ArduinoFiles.h (2 hits) Line 43: class PrintFile : public print_t, public BaseFile { Line 61: class StreamFile : public stream_t, public BaseFile { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\common\FsBlockDeviceInterface.h (1 hit) Line 37: class FsBlockDeviceInterface { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\common\FsCache.h (1 hit) Line 37: class FsCache { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\common\FsName.h (1 hit) Line 38: class FsName { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\DigitalIO\DigitalPin.h (1 hit) Line 297: class DigitalPin { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\DigitalIO\SoftSPI.h (1 hit) Line 49: class SoftSPI { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\ExFatLib\ExFatFile.h (4 hits) Line 40: class ExFatVolume; Line 49: class ExName_t : public FsName { Line 61: class ExFatFile { Line 822: class ExFile : public StreamFile<ExFatFile, uint64_t> { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\ExFatLib\ExFatFormatter.h (1 hit) Line 32: class ExFatFormatter { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\ExFatLib\ExFatPartition.h (2 hits) Line 42: class ExFatFile; Line 61: class ExFatPartition { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\ExFatLib\ExFatVolume.h (1 hit) Line 33: class ExFatVolume : public ExFatPartition { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FatLib\FatFile.h (5 hits) Line 40: class FatVolume; Line 61: class FatLfn_t : public FsName { Line 76: class FatSfn_t { Line 108: class FatFile { Line 1023: class File32 : public StreamFile<FatFile, uint32_t> { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FatLib\FatFormatter.h (1 hit) Line 33: class FatFormatter { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FatLib\FatPartition.h (1 hit) Line 52: class FatPartition { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FatLib\FatVolume.h (1 hit) Line 37: class FatVolume : public FatPartition { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FsLib\FsFile.h (2 hits) Line 39: class FsBaseFile { Line 869: class FsFile : public StreamFile<FsBaseFile, uint64_t> { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FsLib\FsFormatter.h (1 hit) Line 33: class FsFormatter { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\FsLib\FsVolume.h (2 hits) Line 35: class FsFile; Line 40: class FsVolume { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\ArduinoStream.h (2 hits) Line 37: class ArduinoInStream : public ibufstream { Line 107: class ArduinoOutStream : public ostream { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\bufstream.h (2 hits) Line 39: class ibufstream : public istream { Line 94: class obufstream : public ostream { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\fstream.h (4 hits) Line 37: class StreamBaseClass : protected StreamBaseFile, virtual public ios { Line 72: class fstream : public iostream, StreamBaseClass { Line 160: class ifstream : public istream, StreamBaseClass { Line 214: class ofstream : public ostream, StreamBaseClass { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\ios.h (2 hits) Line 50: class ios_base { Line 382: class ios : public ios_base { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\iostream.h (1 hit) Line 156: class iostream : public istream, public ostream {}; C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\istream.h (1 hit) Line 37: class istream : public virtual ios { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\ostream.h (1 hit) Line 37: class ostream : public virtual ios { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\iostream\StdioStream.h (1 hit) Line 113: class StdioStream : private StreamBaseFile { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\MinimumSerial.h (1 hit) Line 37: class MinimumSerial : public print_t { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\RingBuf.h (1 hit) Line 72: class RingBuf : public Print { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdCard\SdCard.h (1 hit) Line 62: class SdCardFactory { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdCard\SdCardInterface.h (1 hit) Line 37: class SdCardInterface : public FsBlockDeviceInterface { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdCard\SdioCard.h (2 hits) Line 41: class SdioConfig { Line 62: class SdioCard : public SdCardInterface { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdCard\SdSpiCard.cpp (1 hit) Line 27: class Timeout { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdCard\SdSpiCard.h (4 hits) Line 71: class SharedSpiCard : public SdCardInterface { Line 73: class SharedSpiCard : public FsBlockDeviceInterface { Line 75: class SharedSpiCard { Line 377: class DedicatedSpiCard : public SharedSpiCard { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SdFat.h (5 hits) Line 50: class SdBase : public Vol { Line 408: class SdFat32 : public SdBase<FatVolume, FatFormatter> { Line 416: class SdExFat : public SdBase<ExFatVolume, ExFatFormatter> { Line 424: class SdFs : public SdBase<FsVolume, FsFormatter> { Line 465: class SdFile : public PrintFile<SdBaseFile> { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SpiDriver\SdSpiArduinoDriver.h (1 hit) Line 39: class SdSpiArduinoDriver { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SpiDriver\SdSpiBareUnoDriver.h (1 hit) Line 92: class SdSpiDriverBareUno { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SpiDriver\SdSpiBaseClass.h (1 hit) Line 35: class SdSpiBaseClass { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SpiDriver\SdSpiDriver.h (3 hits) Line 94: class SdSpiSoftDriver; Line 98: class SdSpiBaseClass; Line 109: class SdSpiConfig { C:\Users\Bill\Documents\ArduinoSdFat\libraries\SdFat\src\SpiDriver\SdSpiSoftDriver.h (2 hits) Line 36: class SdSpiSoftDriver { Line 99: class SoftSpiDriver : public SdSpiSoftDriver {