SdFat Meaning Error Codes on Mount Failed

Hi,

I started having a problem with otherwise very reliable SD Cards. I'm using SanDisk Ultra A1 SD Cards and had no problems at all for a long time now. For a few weeks, I started testing my code on the new ESP32-S3. Here I started to get Write errors after 5-10h of running.

The weird thing is:

the same code/setup works fine with Intenso 8GB Cards for over 24h.
the SD Card work fine on ESP32, Teensy, etc.
When the SD Card reports a Write Error I'm unable to make the SD Card work again until I plug it into a Laptop SD Card reader once. after that, the sd card works again.

This is what i tried:

First, the SD Write fails. here is some info I get from the File Obj. after the write error

[31849169] file.dlog: name="LOG06.TXT", position=491554816, size=491554816, werrors=1, wtime=31847964, isOpen=1
[31849170] file.err: restarting system

1.) The system reboots if a write error is detected to try to just remount the sd. this does not work and i get this at mount

[3495] jFS_SdFat.err: msg=mount failed
[3495] jFS_SdFat.err: code=1, data=0

2.) unplug and replug the sd card without power cycle the Microcontroller, then rebooting the microcontroller per software. result is

[5059] jFS_SdFat.err: msg=mount failed
[5059] jFS_SdFat.err: code=17, data=FF

3.) complete power cycle of microcontroller and SD Card.At this point the sd and microcontroller should be in the state where all works again but i get the same error as 2.)

4.) I unplug sd, put in a sd card reader in windows and the sd is readable no problem.

5.) sd card plugged back into the microcontroller first startup does not work again mount fails. after a second power cycle, the sd works again.

Does this ring a bell ? Is there a lookup table or other info what the code and data values mean at mount error?

I'm doing the 0xFF SPI Sends at boot to reset the sd card. after 10x 0xFF send i still get no 0xFF response. Here the code for reference


static void storage_reset_sd() 
{
	LogD("storage.sd: msg=reset start");

	//reset / unmount sd card
	// https://forum.arduino.cc/t/resolved-how-can-i-reboot-an-sd-card/349700
	pinMode(PIN_SPI_CS_SD, OUTPUT);
	digitalWrite(PIN_SPI_CS_SD, HIGH);
	delay(100);
	
	SPI.beginTransaction(SPISettings(100000, MSBFIRST, SPI_MODE0));
	digitalWrite(PIN_SPI_CS_SD, LOW);
	for(int i = 0; i < 10; i++)
	{
		byte v = SPI.transfer(0xFF);
		if(v == 0xFF)
		{
			LogD("storage.sd: reset_count=%d", i);
			break;
		}
	}
	digitalWrite(PIN_SPI_CS_SD, HIGH);
	SPI.endTransaction();

	LogD("storage.sd: msg=reset end");

}

Jun 20 '22 08:06 savejeff

What SPI clock rate are you using? Errors like this are often due to marginal SPI signals. The SPI driver should limit the clock rate to the max reliable rate but it is common for this to be a bug in SPI drivers. See if a slower clock works.

If an SD has a write error it may take up to 512 + a few SPI clocks to complete the write. The card will not respond to a reset CMD0 until the write is complete. The sector being written will be corrupt.

In the past there have been problems like this with ESP SPI drivers interacting with FreeRTOS handling WiFi. I don't know if the ESP-S3 has a new SPI driver. Old SPI drivers had a semaphore problem that would cause SPI to fail until a reboot.

Jun 20 '22 12:06 greiman

ESP32-S3 has unique code for max SPI speed. An example is in this file:

\Bill\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\cores\esp32\esp32-hal-spi.c

#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
            if(calPre > 0xF) {
                reg.clkdiv_pre = 0xF;
#else
            if(calPre > 0x1FFF) {
                reg.clkdiv_pre = 0x1FFF;
#endif

In fact there are a huge number of #if sections in the driver for ESP-S2/ESP-S3 so expect problems with the new ESP32-S3

 C:\Users\Bill\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\libraries\SPI\src\SPI.cpp (1 hit)
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3
  C:\Users\Bill\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\cores\esp32\esp32-hal-spi.c (54 hits)
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#elif CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#elif CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 || CONFIG_IDF_TARGET_ESP32C3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3
  C:\Users\Bill\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\cores\esp32\esp32-hal-spi.h (1 hit)
#if CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32S3

Jun 20 '22 13:06 greiman

so your guess is that the SPI transaction failes very very infrequent due to too high of a clock rate or internal problems?

I'll try a reduced clock rate. I was getting pretty good write rates but didn't check the actual clock speed for SPI.

About the Corrupted sector, that means I could recover from this by sending up to 512 bytes of 0xFF to "end the spi write" conversation?

Jun 20 '22 13:06 savejeff

You may be able to recover from a write failure. The SD card is looking for a begin token and then will require 512 data plus two checksum bytes. Most cards accept a command while waiting for the begin token. By default cards ignores the checksum in SPI mode unless you select use checksum in SdFatConfig.h to force use. Calculation of a software checksum is slow.

You can generate a list of error codes with this example.

A recent list is here.

Jun 20 '22 13:06 greiman

A thx. Code 1 - Reset Failed is understandable Code 17 - Activate card initialization doesn't mean much to me.

I just looked into my Sd card code. I mount the SD Card with sd.begin(pin_cs, SD_SCK_MHZ(20));. I'll try 10Mhz, but 20Mhz is not even max spi speed. I really hope this is not an SPI Driver issue of the ESP32-S3. Sofar I'm pretty impressed with it.

FYI: I'm using SPI_DRIVER_SELECT 0. might this be caused by the optimized driver for ESP32. I can try SPI_DRIVER_SELECT 1 after the 10Mhz test.

Jun 20 '22 14:06 savejeff

Code 17 is from this code at about line 190 of SdSpiCard.cpp

  // initialize card and send host supports SDHC if SD2
  arg = type() == SD_CARD_TYPE_SD2 ? 0X40000000 : 0;
  timeout.set(SD_INIT_TIMEOUT);
  while (cardAcmd(ACMD41, arg) != R1_READY_STATE) {
    // check for timeout
    if (timeout.timedOut()) {
      error(SD_CARD_ERROR_ACMD41);  // <<----------ErrorCode 0X17 Here
      goto fail;
    }
  }

Here is a description of ACMD41:

SD_SEND_OP_COND (ACMD41) is used to start initialization and to check if the card has completed initialization. It is mandatory to issue CMD8 prior to the first ACMD41. Receiving of CMD8 expands the CMD58 and ACMD41 function; HCS (High Capacity Support) in the argument of ACMD41 and CCS (Card Capacity Status) in the response of CMD58. HCS is ignored by the card, which didn't accept CMD8. Standard Capacity SD Memory Card ignores HCS. The "in idle state" bit in the R1 response of ACMD41 is used by the card to inform the host if initialization of ACMD41 is completed. Setting this bit to "1" indicates that the card is still initializing. Setting this bit to "0" indicates completion of initialization. The host repeatedly issues ACMD41 until this bit is set to "0". The card checks the HCS bit in the OCR only at the first ACMD41. While repeating ACMD41, the host shall not issue another command except CMD0.

The ESP SPI driver is extremely complex and is multi-threaded with WiFi access so is riddled with mutex locks to inter-leaf WiFI and user SPI.

Jun 20 '22 15:06 greiman

Is the WiFi connected over SPI on the ESP32 ?

Could this then be fixed by using the second SPI Port? Is hard to find information in this

Jun 20 '22 16:06 savejeff

You may not be using SPI for WiFi. Looks like ESP-S3 has a different structure so I can't be sure what each of the four SPI ports are used for.

https://docs.espressif.com/projects/esp-idf/en/v4.4/esp32s3/api-reference/peripherals/spi_master.html

Overview of ESP32-S3’s SPI peripherals

ESP32-S3 integrates 4 SPI peripherals.

SPI0 and SPI1 are used internally to access the ESP32-S3’s attached flash memory. Both controllers share the same SPI bus signals, and there is an arbiter to determine which can access the bus.

Currently, SPI Master driver does not support SPI1 bus.

SPI2 and SPI3 are general purpose SPI controllers. They are open to users. SPI2 and SPI3 have independent signal buses with the same respective names. SPI2 has 6 CS lines. SPI3 has 3 CS lines. Each CS line can be used to drive one SPI slave.

Jun 20 '22 17:06 greiman

Looks like ESP-S3 SPI has a bug fix:

There is a newer bugfix release of this ESP-IDF version. The latest bugfix release is v4.4.1

Jun 20 '22 17:06 greiman

Okay that is interesting. Thanks for finding the details! As I'm working with PlatformIO, I'll have to wait for a new firmware release for the espressif core.

I'll report back if usage of SPI2 helps if that is even possible. i think SPI2 is the default first SPI in Arduino and SPI3 the second SPI port in arduino.

EDIT: ok the current version of Arduinos ESP32 Core (2.0.3) already is based on 4.4.1. This is also the Arduino Core version the Lastest PIO Core is based on. Can you give me the source where you found the note about the SPI bugfix ?

Jun 20 '22 17:06 savejeff

Are you using dedicated SPI for the SD card?

Jun 20 '22 18:06 greiman

no im using SPI_DRIVER_SELECT 0 and SHARED_SPI. But i'm currently not using any other slaves on SPI

Dedicated means the SPI can not be used for other slaves right ?

Jun 20 '22 18:06 savejeff

Yes dedicated SPI keeps the SD in read or write mode as long as possible by keeping chip select low. This can provide a major performance improvement for high end cards like your SanDisk Ultra A1 SD Cards.

You could try SPI_DRIVER_SELECT 1. That may much slower.

You could try the combination:

#define SPI_DRIVER_SELECT 1
#define USE_SPI_ARRAY_TRANSFER 1

This works well on some boards.

Jun 20 '22 19:06 greiman

The Logging ran through the night and is now at 16h with no write errors. Seems like switching to 10Mhz fixed the issue for now. I'll let it run to 24h. After that I'll try ou the dedicated SPI Mode as well as SPI_DRIVER_SELECT 1 with 20Mhz. I have pretty short soldered wires to the SD Card module, so noise should to be a problem.

Array Transfer is always on in my code. if this is turned off, is every byte transferred in a new transaction with CS High and Low between it? That must be massively inefficient.

Jun 21 '22 08:06 savejeff

Array transfer, USE_SPI_ARRAY_TRANSFER 1, only applies when using the standard driver, SPI_DRIVER_SELECT 1. Unfortunately the Arduino standard call requires copying data to a temp buffer on transmit and filing the buffer with 0XFF on receive.

void.transfer( uint8_t* buf, size_t size) uses the same buffer for transmit/receive.

The ESP custom driver, SPI_DRIVER_SELECT 0, always uses a special non Arduino call: void transferBytes(const uint8_t * data, uint8_t * out, uint32_t size);

Jun 21 '22 12:06 greiman

I put a scope on an ESP32S2 Feather. I used this loop for a signal.

void loop() {
  delayMicroseconds(100);
  SPI.transfer(0XAA);
}

Here are the signals for a 20MHz clock. The overshoot is due to inductance in a bit of ground wire for the scope. Probably not really in the signal. SCK ESP32S2CLK MOSI ESP32S2MOSI

These are great. If your ESP32-S3 has signals like this it should run fine.

The clock even looks good at max speed of 40MHz. ESP32S2_40MHz

I used these settings SPI.beginTransaction(SPISettings(50000000, MSBFIRST, SPI_MODE0)); The ESP32S2 SPI driver selected 40MHz.

Jun 21 '22 15:06 greiman

I'll try to reproduce this with my oscii tomorrow but I'm afraid that this will be too high of a frequency for mine. I'll report back the results.

The Experiment with SPI_DRIVER_SELECT 1 USE_SPI_ARRAY_TRANSFER 1 with 20Mhz and the dedicated spi is running now. let's see if it still runs tomorrow morning. currently, I'm getting around 700 kbyte/s on the writes. that's pretty much what i usually get. I'm doing 2024 bytes writes with no is_busy check writing to two files inter-leaved.

Jun 21 '22 19:06 savejeff

Array transfer, USE_SPI_ARRAY_TRANSFER 1, only applies when using the standard driver, SPI_DRIVER_SELECT 1. Unfortunately the Arduino standard call requires copying data to a temp buffer on transmit and filing the buffer with 0XFF on receive.

void.transfer( uint8_t* buf, size_t size) uses the same buffer for transmit/receive.

The ESP custom driver, SPI_DRIVER_SELECT 0, always uses a special non Arduino call: void transferBytes(const uint8_t * data, uint8_t * out, uint32_t size);

why does the standard driver (aka the Arduino SPI ) work with USE_SPI_ARRAY_TRANSFER and not with the specialized one? I would have expected this the other way around

Jun 21 '22 19:06 savejeff

The optimizes driver always uses void transferBytes(const uint8_t * data, uint8_t * out, uint32_t size); which is better than the Arduino array transfer. no copy or fill is required

To send a 512 byte block it uses:

  SPI.transferBytes(null, buffer, 512);

To receive a 512 byte block it uses: SPI.transferBytes(buffer, null, 512);

The standard Arduino library driver has the USE_SPI_ARRAY_TRANSFER option because it uses a 512 byte temporary buffer and on for many boards such as Uno it is slower than the byte-at-a-time loop since the array transfer call just has a byte-at-a-time loop.

Here is the code for the Arduino library driver receive and send:

inline uint8_t SdSpiArduinoDriver::receive(uint8_t* buf, size_t count) {
#if USE_SPI_ARRAY_TRANSFER
  memset(buf, 0XFF, count);
  m_spi->transfer(buf, count);
#else  // USE_SPI_ARRAY_TRANSFER
  for (size_t i = 0; i < count; i++) {
    buf[i] = m_spi->transfer(0XFF);
  }
#endif  // USE_SPI_ARRAY_TRANSFER
  return 0;
}

// limited to 512 bytes max
inline void SdSpiArduinoDriver::send(const uint8_t* buf, size_t count) {
#if USE_SPI_ARRAY_TRANSFER
  if (count <= 512) {
    uint8_t tmp[512];
    memcpy(tmp, buf, count);
    m_spi->transfer(tmp, count);
  }
#else  // USE_SPI_ARRAY_TRANSFER
  for (size_t i = 0; i < count; i++) {
    m_spi->transfer(buf[i]);
  }
#endif  // USE_SPI_ARRAY_TRANSFER
}

Jun 22 '22 12:06 greiman

Ah okay, i think i start to understand how the SPI communication works. I really like how versatile the SdFat lib is built with many options to adjust to different needs. I'm using it for some time now and i am slowly building up an understanding of how it works.

We previously talk on a other issue about connecting SdFat to a USB Connection. A few weeks ago i played around with the hardware USB of the ESP32-S3. using the TinyUSB lib I emulated a FAT16 Medium that virtually mounted selected files from a connected Sd Card. later i thought about that conversation and figured the SdFat lib might have been useful in the emulation of the FAT file system. Is there an abstraction that just does the sector reads and writes? then a could have routed the sector reads and writes coming from TinyUSB to the SdFat lib.

About the SD Card Write Tests, the system ran no problem for 24h with DRIVER_SELECT 1, ARRAY TRANSFER and 20Mhz as well as dedicated driver. Next ill try the same setup but with DRIVER_SELECT 0. Due to other work I did not yet had the time for the Oscii Stuff. maybe tomorrow

EDIT: The Test didn't make it through the night. Took only 2h for a write to fail. Reboot/Remount and Reset with up to 512 xFF did not work. tried it twice. After a power cycle the sd mounted fine again (without the use of windows card reader) Im repeating this to make sure, but considering that the exact same setup with DRIVER SELECT 1 ran for solid 24h it seems pretty sure that there is something going wrong with the specialized driver

Jun 22 '22 20:06 savejeff

Second Run ran for 19h before crashing. My Monitoring Script crashed as well, so I'm not 100% sure it was not due to something restarting on the serial port. So its not as cut and dry as i wish it would be. I'll start another test run with DRIVER_SELECT 1, ARRAY TRANSFER and 20Mhz as well as dedicated driver to make sure that config works reliably.

Jun 24 '22 11:06 savejeff

SdFat SdFat copied to clipboard

Meaning Error Codes on Mount Failed

SdFat
SdFat copied to clipboard