SdFat v2 not working on Particle Devices

[LukeL99]

I'm attempting to get v2.2.0 of SdFat working on a Particle B402/404 device. I've had success getting v1.X to work, but I haven't had any luck with V2+.

Here is a bare bones sketch, which I am compiling via Particle Workbench, targeting deviceOS v4.0.2. It fails at the compilation stage without building a valid binary.

#define ENABLE_DEDICATED_SPI 1

#include <SPI.h>
#include <SdFat.h>

void setup() {
  SdSpiConfig config(8, DEDICATED_SPI, SPI_FULL_SPEED, &SPI);
  SdFat sd;
  sd.begin(config);
}

void loop() {
  os_thread_yield();
}

Output gives both a warning and an error:

In file included from c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatVolume.h:27,
                 from c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatLib.h:27,
                 from c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatName.cpp:28:
c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatFile.h:124:5: warning: "FNAME_FLAG_NEED_LFN" is not defined, evaluates to 0 [-Wundef]
  124 | #if FNAME_FLAG_NEED_LFN & (FAT_CASE_LC_BASE || FAT_CASE_LC_EXT)
      |     ^~~~~~~~~~~~~~~~~~~
c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatFile.h:124:28: warning: "FAT_CASE_LC_BASE" is not defined, evaluates to 0 [-Wundef]
  124 | #if FNAME_FLAG_NEED_LFN & (FAT_CASE_LC_BASE || FAT_CASE_LC_EXT)
      |                            ^~~~~~~~~~~~~~~~
c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatFile.h:124:48: warning: "FAT_CASE_LC_EXT" is not defined, evaluates to 0 [-Wundef]
  124 | #if FNAME_FLAG_NEED_LFN & (FAT_CASE_LC_BASE || FAT_CASE_LC_EXT)
      |                                                ^~~~~~~~~~~~~~~
c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatName.cpp: In member function 'size_t FatFile::printName8(print_t*)':
c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatName.cpp:337:22: error: invalid conversion from 'char*' to 'const uint8_t*' {aka 'const unsigned char*'} [-fpermissive]      
  337 |       n += pr->write(buf, str - buf);
      |                      ^~~
      |                      |
      |                      char*
In file included from ../wiring/inc/spark_wiring_string.h:34,
                 from ../wiring/inc/spark_wiring_stream.h:30,
                 from ../wiring/inc/spark_wiring.h:40,
                 from ./inc/application.h:42,
                 from ./inc/Particle.h:5,
                 from ./inc/Arduino.h:11,
                 from c:\users\lukel\onedrive\documents\development\particle-workbench\sdfat-v2-test\lib\sdfat\src\sdfatconfig.h:57,
                 from c:\users\lukel\onedrive\documents\development\particle-workbench\sdfat-v2-test\lib\sdfat\src\common\SysCall.h:33,
                 from c:\users\lukel\onedrive\documents\development\particle-workbench\sdfat-v2-test\lib\sdfat\src\common\debugmacros.h:27,
                 from c:/Users/lukel/OneDrive/Documents/development/particle-workbench/sdfat-v2-test/lib/SdFat/src/FatLib/FatName.cpp:26:
../wiring/inc/spark_wiring_print.h:78:41: note:   initializing argument 1 of 'virtual size_t Print::write(const uint8_t*, size_t)'
   78 |     virtual size_t write(const uint8_t *buffer, size_t size);
      |                          ~~~~~~~~~~~~~~~^~~~~~
make[3]: *** [../build/module.mk:274: ../build/target/user/platform-23-m/sdfat-v2-test/SdFat/src/FatLib/FatName.o] Error 1
make[2]: *** [../../../build/recurse.mk:12: user] Error 2
make[1]: *** [../build/recurse.mk:12: modules/boron/user-part] Error 2
make: *** [C:\Users\lukel\.particle\toolchains\buildscripts\1.11.0\Makefile:68: compile-user] Error 2

I tried the SdFat-beta repo with the same errors. If I can provide any additional info please let me know.

[greiman]

Try removing FatFile.h lines 124-126 here. They should not be there since the symbols have been changed from defines to const uint8_t.

The error at line 337 of FatName.cpp is due to an incompatibility of Print::write().

Arduino has the following in Print.h so char* works.

    size_t write(const char *buffer, size_t size) {
      return write((const uint8_t *)buffer, size);
    }

Try changing line 337 of FatName.cpp here to: n += pr->write(reinterpret_cast<uint8_t*>(buf), str - buf);

Let me know if the above works.

[LukeL99]

It looks like it worked! At least it compiled. I'm in the middle of the conversion from V1 to V2 API, so I'll update here if I have issues along the way.

I also had to change line 221 of ExFatFilePrint.cpp.

I also want to say thank you for maintaining such a massive and important library. Open source developers don't get the credit they deserve, and to be still working on it for 10+ years is very impressive. Thanks for your help.

[LukeL99]

I'm running into a lot of issues with V2.x on Gen 3 Particle devices, even after I change the above and the code compiles.

With the 2.x version of the library, the device becomes unresponsive to both cellular commands and to being put into DFU mode via the Serial port. The only thing I can do is a power cycle of the device to make it responsive again.

If I put a logging statement into the main loop, I can see that that is still running. I have a software timer blinking an LED, and that is still operating as expected.

Is it possible that this library overwrites some system functions that are being used to communicate with the UBlox module and/or the DFU via Serial? I'm kind of stumped at the moment.

I'm going to try to put together a minimal example and I'll post it here.

[greiman]

The most common hang problem is using dedicated SPI.

What configuration are you using in sd.begin() ?

[LukeL99]

SdSpiConfig config(8, DEDICATED_SPI, SPI_FULL_SPEED);

Try changing DEDICATED_SPI to SHARED_SPI.

SPI_FULL_SPEED is 50MHz. Try replacing it with a slower speed. Try:

SdSpiConfig config(8, SHARED_SPI, 4000000)

If that works try faster speeds.

[greiman]

I accidentally edited your post adding this:

Try changing DEDICATED_SPI to SHARED_SPI.

SPI_FULL_SPEED is 50MHz. Try replacing it with a slower speed. Try:

SdSpiConfig config(8, SHARED_SPI, 4000000)

If that works try faster speeds.

[LukeL99]

That seems to have helped a lot! I'm still testing but so far it's much more stable.

One of the reasons I'm working on updating the SD card library is to increase reliability of our SD card data writing. We had a scenario where some of our files were not writing to the SD card (after 1+ week of continuous operation), but we didn't have a reliable way to A) test whether our writes were successful or B) recovering reliably when we had an issue.

We've implemented a series of functions to increase reliability. Can you give me your opinion on this implementation?

bool StorageManager::initSD()
{
    SdFile::dateTimeCallback([](uint16_t *date, uint16_t *time) {
        // return date using FAT_DATE macro to format fields
        *date = FAT_DATE(Time.year(), Time.month(), Time.day());
        // return time using FAT_TIME macro to format fields
        *time = FAT_TIME(Time.hour(), Time.minute(), Time.second());
    });

    SdSpiConfig config(SD_CS_PIN, SHARED_SPI, SPI_HALF_SPEED);

    if (!sd.begin(config))
    {
        Log.error("Unable to open SD card");
        return false;
    }
    else
    {
        Log.info("SD card opened");
        return true;
    }
}

bool StorageManager::resetSDConnection()
{
    sd.end();
    delay(50);
    bool result = initSD();
    if (result)
    {
        Log.info("Reset SD successfully.");
        PublishQueuePosix::instance().publish("sd/v1", "Reset SD successfully.", PRIVATE | WITH_ACK);
    }
    else
    {
        Log.error("Unable to reset SD.");
        PublishQueuePosix::instance().publish("sd/v1", "Unable to reset SD.", PRIVATE | WITH_ACK);
    }
    return result;
}

bool StorageManager::testAndResetSDConnection()
{
    if (!sdWriteCheck())
    {
        return resetSDConnection();
    }
    return true;
}

bool StorageManager::sdWriteCheck()
{
    File32 testFile;
    if (!(testFile = sd.open("/fs_test.txt", O_WRITE | O_CREAT)))
    {
        Log.error("Unable to access root of FS.");
        return false;
    }
    testFile.close();
    sd.remove("/fs_test.txt");
    return true;
}

On every operation that writes a new file to the SD card (every ~10 minutes), we call testAndResetSDConnection(). This seems to account for scenarios where the physical connection to the SD card is removed and the replaced, but it seems like a lot of overhead to test the SD state and recover it. Do you have a best practice for fault tolerance that you've used in the past?

Thanks again!

[greiman]

I can't help much with reliability.

Key hardware things are to have short wires to the SD socket and pullups on all data pins. Sometimes bigger caps on power help, SD cards draw lots in pulses when programming flash. Anything you can do to avoid noise from Wi-Fi or Cell radios will help.

Only use shared SPI if you need speed. If you only want bursts of speed you can use bool setDedicatedSpi(bool value)

  sd.setDedicatedSpi(true);  // Should check return for error
 //  do high performance access
  sd.setDedicatedSpi(false); // Should check return for error

Dedicated SPI mode keeps the SD selected and in extended multi-block read or write mode.

Slower SPI speeds are more reliable on many chips. Lots of chips don't have sufficient drive to produce clean square signals.

You can call file.sync() to avoid corruption if the SD is removed, power fails, or other glitch happens. bool sync(); causes the directory entry to be updated and all cached data to be written to the SD.

It is the same as closing the file and reopening the file but very efficient.

[LukeL99]

Excellent. Thanks for the help and advice!