lfs is erasing a block on every append

[keck-in-space]

Hello,

I am in the process of integrating lfs into my project and I am able to write to a file without too much of an issue, but I am noticing some odd behavior.

Every time I append to a file, an entire block is erased. This is a bit odd since my block size is enormous at 256KB. That's the minimum erase size of the chip, I'm only writing 321 bytes at a time, and the file size is around 47KB... so no additional blocks should be needed unless I'm missing something.

My configuration is below. I'm using the Cypress S25FL512S.

#define LFS_CACHE_SIZE 512U

uint8_t readBuff[LFS_CACHE_SIZE];
uint8_t writeBuff[LFS_CACHE_SIZE];
uint8_t lookBuff[LFS_CACHE_SIZE];
uint8_t fileBuff[LFS_CACHE_SIZE];

lfs_t lfs;
const struct lfs_config cfg = {
    // block device operations
    .read = &lfs_flash_read,
    .prog = &lfs_flash_write,
    .erase = &lfs_flash_erase,
    .sync = &lfs_flash_sync,

    // block device configuration
    .read_size = 16,
    .prog_size = 16,
    .block_size = 256 * 1024,
    .block_count = (64 * 1024 * 1024) / (256 * 1024),
    .block_cycles = 500,
    .cache_size = LFS_CACHE_SIZE,
    .lookahead_size = LFS_CACHE_SIZE,

    .read_buffer = readBuff,
    .prog_buffer = writeBuff,
    .lookahead_buffer = lookBuff,
};

const struct lfs_file_config file_cfg = {
	.buffer = &fileBuff,
	.attr_count = 0,
};

[geky]

Hi @keck-in-space, thanks for creating an issue.

This sounds like it may be related to https://github.com/ARMmbed/littlefs/issues/344 and https://github.com/ARMmbed/littlefs/issues/368.

There are some issues around littlefs not knowing if the end block in a file is erased. If you write a file in one pass (open->write->write->write->close), then it should perform the minimum number of erases. But there may be problems if other operations mess with littlefs's knowledge of the state of the file.

• Extra seeks (open->write->seek->write->close)

  If these don't change the file position, then we shouldn't need to do an extra erase. However I'm not sure littlefs is smart enough to avoid that at the moment. @Johnxjj has pointed out it should be possible to avoid the extra erase in more cases.

• Extra syncs (open->write->sync->write->close)

  This should be easy for littlefs to avoid extra erases since syncs will never change the erase state of the underlying block. littlefs should be handling this but it's possible there's a bug, it would at least be worth checking.

Is it possible to remove seek or sync calls between write operations to see if either of those reduces erases?

[keck-in-space]

Hi @geky! Thanks for the quick response. I was opening and closing the file with each write... I'll see what kind of performance I see if I perform the writes with an open file and get back to you.

[geky]

Ah yes, that would cause the extra erases. littlefs doesn't know if an opened file has been erased, so it is conservative and always erases the end block. We can't rely on the erase value of an "erased" block since it's not the same on all storage (an encrypted block device may represent erased as 1s passed through the decryption function, for example).

I've thought of adding a flag to file's metadata that indicates if the end block is erased, but we would need to update it before and after we write to the file. Once we've started writing, even if we lose power, we don't want to accidentally try to write the same location again.

[keck-in-space]

Just wanted to follow up and say you're correct that leaving the file open between writes prevents the extra erase cycles from occurring. Feel free to leave this open if you want to track it for an optimization task, but for my case, the issue has been solved. Thank you!

[keck-in-space]

Actually, I'm noticing the same issue with syncs... Every time I sync after write, the erase function occurs and a new block is allocated... I'm using the latest commit to master.

[geky]

Yeah, this is the issue here https://github.com/ARMmbed/littlefs/issues/344, described in more detail here https://github.com/ARMmbed/littlefs/issues/344#issuecomment-567195031

There's a possible optimization that can be done if your file's size is aligned to your prog size, but it's not possible otherwise because padding gets written out.

Fixing this may need a more general and complicated solution.

[keck-in-space]

Hmm. So the flash chip we're using can do 1 byte size read/writes, but if I'm reading your comment correctly, that won't yet solve the issue with a large block size. Also, if I'm reading correctly, the entire file will be lost if a sync or close does not occur before being reset.

This is a pretty big issue for us since the block size of 256KB means we will have a very long wait time if we request a sync and then return to write on every call to store log data.

[geky]

Yes that is correct.

It's interesting this is the first chip I've seen that has both the large (>4KiB) erase size and the byte-level writes. Usually large erases go hand-in-hand with larger writes, which makes the easy optimization not work. But you're right this looks like the case for the S25FL512S part.

Unfortunately this will at least need to wait for https://github.com/ARMmbed/littlefs/pull/372 before I personally can look into implementing a solution. It's been pointed out in a few places that this is a problem for logging use cases.

[keck-in-space]

Thanks so much for this reply. It helps a lot with understanding the trade-offs for the various flash chip configurations and the way LFS interacts with them. Do you see any optimization issues with chips that have a 4, 8, or 16KB erase size?

[geky]

It's hard to say because sometimes an erase takes too long for an application and sometimes it doesn't.

4KiB erases should be ~64x faster than 256KiB erases (assuming the same underlying hardware, which is likely not true). It would still be a big improvement to reuse erases while appending, but it may not be needed if the erase time is acceptable.

[sslupsky]

@geky I think spi nand devices commonly have large erase size and byte level writes. I am using a Toshiba device where the page size is 4096 bytes and blocks are 256 Kbytes. Erase size is the block size. However, I can write 1 byte at a time to the page because the page is cached in a buffer on the nand chip until the nand chip receives a command to write the on-chip page buffer to the memory array.

I am finding that appending a log file is not what I expected. I think I am observing that every time there is a sync, the entire file is read and then re-written. When the log file was a few 100 bytes this did not really seem to be a problem. But, as the file has grown in size, I have noticed significant periods of time when the file system is doing something.

My current method of recording the log is to open the file and then periodically write and sync. So, open-write-write-sync-write-sync- ...

I do not expect to close the file because it is the system log I am writing to. So, this file remains open as long as the device is turned on and it receives the log file. I may occasionally open the file a second time using the shell to tail it or otherwise read or inspect the file. In that case, the second open is closed (but the logging file handle remains open).

[andyp1per]

Reading these issues is leading me to believe that littlefs is totally unsuited to logging - which is a pain, because it has lots of other great features. The issues seem to be that writes are expensive because erase is done on demand rather than on format - I would much rather pay the cost of erasing on format than on demand while I am logging. The other issue is that sync is expensive, but needed because of #344 - it's vastly too expensive to be called per-block I am running on an H7 with nand ram and writes/syncs are optimized to only happen on block boundaries and it still costs 25% CPU, which is insane. The fact that you have to call a very expensive function just to see any data at all feels like a design flaw - yes 0 bytes is consistent but not useful in the slightest. I'd love to know whether there are workarounds but its sounding more and more like littlefs is just not fit for purpose in this case ☹️

The bespoke implementation I am looking at replacing erases the entire chip on format and then does append-only of file blocks, each block containing enough information to know if its the last block written. It's very fast and has none of these issues, but it also doesn't have all the nice features of a file system - really just want to get both.

[andyp1per]

There's a possible optimization that can be done if your file's size is aligned to your prog size, but it's not possible otherwise because padding gets written out.

We already do this, so its not a problem for us, but I'm struggling to see how this can be made to help.

[andyp1per]

Just as an example - the W25N01GV has page (2k) program time of 250us and block (128k) erase time of 2ms. So clearly erasing a block per page is a non-starter, even if erases are optimized to be only per-block 2ms is a long time for a real-time system - which many of these devices are used in.

I suppose it begs the question - should I only try and call sync every 128k of data or is this supposed to be optimized away anyway. Typically on these chips we get about 230kB/s writes which means a 4ms delay every second - which really is hopeless. Can't see how it will work without up-front erases.

[andyp1per]

@geky if I am writing consecutive blocks is there a way to anticipate that the next block is unused and maybe pre-erase ahead of time? At least on the Nand flash the erase completes asynchronously, so if there is a way not to wait for it that might improve things.

[geky]

The bespoke implementation I am looking at replacing erases the entire chip on format and then does append-only of file blocks, each block containing enough information to know if its the last block written. It's very fast and has none of these issues, but it also doesn't have all the nice features of a file system - really just want to get both.

There's some joke in here about wanting a high-speed minivan...

@geky if I am writing consecutive blocks is there a way to anticipate that the next block is unused and maybe pre-erase ahead of time? At least on the Nand flash the erase completes asynchronously, so if there is a way not to wait for it that might improve things.

We would need somewhere to store which blocks are erased. There are plans to add a block map which can store this, but it is a work-in-progress.

[andyp1per]

There's some joke in here about wanting a high-speed minivan...

Unfortunately the implementation I am trying to replace is buggy and costs flash, so using littlefs exclusively has huge benefits. Anyway, as you have seen from my other comments and PR I seem to have arrived at an implementation that is fast enough to consider using littlefs exclusively.

If your minivan can do 80 on the motorway then it probably is good enough...

[geky]

If your minivan can do 80 on the motorway then it probably is good enough...

Good point!