Use SD card as store&forward memory

[caveman99]

Not 100% sure if this is a good idea.

[caveman99]

n.b. this enables store&forward as a client for non-esp32 devices.

[markbirss]

@caveman99 The rak board has addon sdcard board also

[caveman99]

@markbirss yes, I wanna go slowly step by step. For the rak I thought the flash add-ons would be a better choice. Still looking into this.

[GUVWAF]

Sounds really nice.

Any idea on how long a write and read takes (since it seems this isn't asynchronous)?

[caveman99]

Any idea on how long a write and read takes (since it seems this isn't asynchronous)?

Nope, i couldn't test it (yet). It's 200ish byte per file, but also the SDcard interface is usually single lane.

[Woutvstk]

Hi I had made a pull request to add hardware support for the NRF52 platform. Someone mentioned that I should add my commits to this pull request. Could you give me acces to this PR branch or would you like to do things differently? Let me know, thanks

[GUVWAF]

Please see #6121: currently this causes the radio to be not detected, so the reorganization of the SPI definitions when using an SD card has to be revisited.

[Woutvstk]

Please see #6121: currently this causes the radio to be not detected, so the reorganization of the SPI definitions when using an SD card has to be revisited.

~~-> Should be fixed with #6126~~ -> Should be fixed with #6132

[Woutvstk]

Since f7feea6, I've been having trouble at runtime on rak4631 in de S&F branch. There is some critical error during initialisation. Something to do with an SPI lock. Nevermind, was an easier fix than I expected. pr #6140

[Woutvstk]

I just added a pull request (#6382) to this branch. The S&F functionality is working on the nRF52 now. (only tested with rak4631+rak15002). However, I ran into some problems:

• The scanning for free space now also works on the nrf52 platform but it takes considerable time (during boot). On the rak4631+RAK15002, it now takes about 8 seconds per GB of SD card to scan the free clusters. After that, free cluster count is maintained, at no further delays are present. This now causes the boot process to be stuck while this scanning is happening. Is this expected behaviour (is this also present on the ESP32?) and could/should this be solved?

• Now, for every packet received, a new file is created. When doing some benchmarks with this new library, write times kept increasing with the increasing file count on the FAT filesystem. Again, is this expected behaviour or could this be a bug? Writing a single historyStruct (~253 bytes) takes different amounts of time depending on how it is written to the SD card. - Appending to existing file: 11ms (not currently done) - Creating new file on empty (formatted) filesystem: 50-70ms - When 200 or so files are already present on the filesystem, creating and writing a new file takes up 150-250ms - If this is because of the FAT filesystem, this could be a problem when scaling

• Also, now only FAT32 is supported (exFat is not) because this reduces flash usage and should provide plenty of storage space.

[markbirss]

@Woutvstk i assume the SdFormatter example should still work ? (i might want to try sdnand and might need to erase it) https://www.adafruit.com/product/4899

[Woutvstk]

@Woutvstk i assume the SdFormatter example should still work ? (i might want to try sdnand and might need to erase it) https://www.adafruit.com/product/4899

Not sure I'm following. The SdCardFormatter example from the SdFat library uses the SdFat library. Meshtastic now uses a wrapper of the SdFat library with another API (similar to the one from the ESP32 SD library). The format function is not included in the wrapper library (SdFatWrapper25) yet. (only functions used in meshtastic are defined for now).

[markbirss]

@Woutvstk i assume the SdFormatter example should still work ? (i might want to try sdnand and might need to erase it) https://www.adafruit.com/product/4899

Not sure I'm following. The SdCardFormatter example from the SdFat library uses the SdFat library. Meshtastic now uses a wrapper of the SdFat library with another API (similar to the one from the ESP32 SD library). The format function is not included in the wrapper library (SdFatWrapper25) yet. (only functions used in meshtastic are defined for now).

Ok, understood, thank you

[caveman99]

maybe we can use the logging class from device-UI instead of doing one file for each message. iirc. @mverch67 created it with the idea in mind to make it re-usable.

[d4rkmen]

hi, this is great feature, indeed, storing data in (PS)RAM is not fail safe. But in this current implementation need to add storing the index value to SD card too. This way we will not loose the entire queue in case of restart (witch is happen very often, like on settings change or exceptions)

[d4rkmen]

I was looking deep inside this branch, and it requires next changes:

• refactor to use custom storage types. possible cases: SDCard / NVS / SPIFFS / USB drive (may be even more ways)
• move to selectable S&F storage type on build, device can have both PSRAM / SDCARD / etc
• getter/setter for node last index - important to keep indeces in non-volatile memory to keep the queue on reboots (settings changes requires reboot and loosing the queue)
• consider FAT has limit of files in a directory

I have made some changes on current (2.6.7) version locally but not sure how to contribute it right.

[d4rkmen]

Consider the FATFS library used for SDCard eats 30kb of RAM, making ESP32 system unusable (api server has no free ram to serve)