support for storage devices > 2 GiB

[benpicco]

Hi, FlashDB uses a signed long as the offset to address the underlying storage media. On a 32 bit ARM MCU this gives a range of ± 2 GiB which is fine for internal and NOR Flash, but NAND Flash and SD cards come in much larger sizes, leaving much of the space unusable.

There are two possible solutions:

• use uint64_t for the offset. This is the easiest solution, but requires 64 bit arithmetic
• use a block, offset combination. This maps most naturally to Flash devices, but requires a larger code / API change.

What do you think would be the best path forward? When logging a lot of data over a long time, 2 GiB are not enough.

(Ok it is possible to use a file system between FlashDB and the storage, but this comes with a performance penalty and more ROM cost)

[armink]

Hi, @benpicco . Thx for your report.

Using 64bit address is a large change for FlashDB. Maybe it should be improved in the next major version (V2.0). And the 64bit address is optional features.

Do you have a requirement for 64bit address in your actual product? What is the usage scenario?

[benpicco]

We are logging ~100kb/s of data to an SD card, so that adds up to exceed 2 GiB after 6h. The problem is that FlashDB also takes longer to seek to the end of the DB the bigger it gets, so this might not work at all. Right now we are just streaming to a file on a FAT fs where we keep the FD open (no DB, just dumping data to a file).

When changing the API I would suggest for the storage callbacks to allow for supplying a user defined void * so multiple backend instances can use the same implementation.

I ported FlashDB to RIOT-OS (https://github.com/RIOT-OS/RIOT/pull/17612) which has it's own flash/memory abstraction (MTD), so I had to write a MTD <-> FAL adapter. This only allows to use a single MTD device for FlashDB as the function callbacks are

int read(long offset, uint8_t *buf, size_t size);

instead of

int read(void *ctx, long offset, uint8_t *buf, size_t size);

where ctx would be set in fal_flash_dev

[armink]

We are logging ~100kb/s of data to an SD card, so that adds up to exceed 2 GiB after 6h. The problem is that FlashDB also takes longer to seek to the end of the DB the bigger it gets, so this might not work at all. Right now we are just streaming to a file on a FAT fs where we keep the FD open (no DB, just dumping data to a file).

Okey. It seems that FlashDB V2.0 needs to be started as soon as possible.

When changing the API I would suggest for the storage callbacks to allow for supplying a user defined void * so multiple backend instances can use the same implementation.

For more MTD device. FAL can support multiple flash device. You can try to register multiple flash devices in FAL and reference them in the partition table.

[benpicco]

For more MTD device. FAL can support multiple flash device. You can try to register multiple flash devices in FAL and reference them in the partition table.

Sure, but as far as I understood those must be different implementations. On RIOT MTD will already abstract away all the different storage backends, so it needs some pointer to differentiate them.

The same can be true for FAL, e.g. what if you have two AT24xxx EEEPROM devices connected that use the same driver/backend?

[armink]

The same can be true for FAL, e.g. what if you have two AT24xxx EEEPROM devices connected that use the same driver/backend?

Maybe FlashDB can support flash backend hook for these function.

fdb_err_t _fdb_flash_read(fdb_db_t db, uint32_t addr, void *buf, size_t size);
fdb_err_t _fdb_flash_erase(fdb_db_t db, uint32_t addr, size_t size);
fdb_err_t _fdb_flash_write(fdb_db_t db, uint32_t addr, const void *buf, size_t size, bool sync);

Then you can call fdb_low_lvl_set_flash_hook(read, wirte, erase) for set RIOT MTD device as storage backend.