Error when writing large files with chunks smaller than 256 bytes

[vad32m]

I've just integrated SPIFFS into my project, target MCU is stm32f407 and memory IC is W25Q32. My application requires working with large (up to 448 kB) files. I was testd SPIFFS with 10kB files and everything seemed to be hunky-dory. But then I switched to testing my app with almost 32kB files and now I am facing following issue: if I write this file with chunks less than 256 bytes, some data loss occur. I cant disclose all the code due to some limitations but here is self-explanatory snippet:

 SPIFFS_gc(&fs, 32768);
  // create a file, delete previous if it already exists, and open it for reading and writing
  fd = SPIFFS_open(&fs, "/update/firmware", SPIFFS_CREAT | SPIFFS_TRUNC | SPIFFS_WRONLY, 0);
  if (fd < 0) {
    printf("errno %i\n", SPIFFS_errno(&fs));
    return 0;
  }
	
  int32_t firmware_size = 0;
  const int32_t one_burst_write_count = 256;
	while (firmware_size < received){
		  int32_t read_count = received - firmware_size;
		  if (read_count > one_burst_write_count){
				  read_count = one_burst_write_count;
			}
                        if (SPIFFS_write(&fs, fd, &firmware_buf[firmware_size],read_count) < 0){
				 printf("write errno %d\r\n", SPIFFS_errno(&fs));
			}
      firmware_size += read_count;
  }
  
  if (SPIFFS_close(&fs, fd) < 0) {
    printf("errno %i\n", SPIFFS_errno(&fs));
    return 0;
  }
  
  fd = SPIFFS_open(&fs, "/update/firmware", SPIFFS_RDONLY, 0);
  if (fd < 0) {
    printf("errno %i\n", SPIFFS_errno(&fs));
    return 0;
  }
  
  printf("firmware file \r\n");
  firmware_size = 0;
  //readout file and check it
  while (!SPIFFS_eof(&fs,fd)){
      SPIFFS_read(&fs, fd, (u8_t *)buf, 1);
      if (buf[0] != firmware_buf[firmware_size]){
          printf("%x error %d != %d\r\n", firmware_size, buf[0], firmware_buf[firmware_size]);
      }
      firmware_size++;
  }
  
  printf("filesize %d\r\n",firmware_size);
  
  if (SPIFFS_close(&fs, fd) < 0) {
    printf("errno %i\n", SPIFFS_errno(&fs));
    return 0;
  }

It works OK when one_burst_write_count is 256 or greater, but if I set it to number, less than 256, SPIFFs wont properly write (or read) file. When it will perform file checks errors will appear, data in file beginning from some address will not correspond to data from the firmware_buf, which is actually source for file. Errors with text like 6774 error 135 != 143 will appear (see printf statement) and file size calculated with firmware_size variable will be little lower. For example file size, when one_burst_write_count = 132 is 0x6777 and file size is 31855 bytes instead required 31900. If I dump that file binary into console it seems that everything is OK, except missing 45 (31900 - 31855) bytes, strting at 0x6777 file offset. If I modify chunk size to 255 errors will still persist, but they will start from offset 0x67F2 and file size will be 31735 bytes. Setting chunk size to 256 fixes the problem. Can anyone approve that this is bug, or maybe I made mistake somewhere? BTW firmware_buf is initialized with binary content, you may found in archive. firmware.zip

[pellepl]

Hi, thanks for a detailed report. I'll look it through more thoroughly when I've got some more time at hand. However, this really should work and have been tested. Similar problems before have in 90 perc of all cases turned out to be a bug in the spiflash driver or in the integration. In meantime, I'd recommend you to really scrutinize your spi flash driver code and make yourself absolute certain that it does what it should. Also, if you're running in a threaded context, make sure the mutex locking works.

Cheers /P

[pellepl]

Oh, and what's your config?

[vad32m]

@pellepl It is quite possible that there are errors in my implementation. I attached archive with my files, so you can see SPIFFS config as well, as driver implementation. And here is one more snippet, that performs initialization

spiflash_t winbond_flash_handler;
static spiffs fs;

#define LOG_PAGE_SIZE       256

static u8_t spiffs_work_buf[LOG_PAGE_SIZE*2];
static u8_t spiffs_fds[32*4];
static u8_t spiffs_cache_buf[(LOG_PAGE_SIZE+32)*4];

static s32_t my_spiffs_read(u32_t addr, u32_t size, u8_t *dst) {
    return SPIFLASH_read(&winbond_flash_handler, addr, size, dst);
 }

static s32_t my_spiffs_write(u32_t addr, u32_t size, u8_t *src) {
    return SPIFLASH_write(&winbond_flash_handler,addr, size, src);
}

static s32_t my_spiffs_erase(u32_t addr, u32_t size) {
    return SPIFLASH_erase(&winbond_flash_handler, addr, size);
} 
  
 
void my_spiffs_mount() {
    spiffs_config cfg;
    
    cfg.hal_read_f = my_spiffs_read;
    cfg.hal_write_f = my_spiffs_write;
    cfg.hal_erase_f = my_spiffs_erase;
    
    int res = SPIFFS_mount(&fs,
      &cfg,
      spiffs_work_buf,
      spiffs_fds,
      sizeof(spiffs_fds),
      spiffs_cache_buf,
      sizeof(spiffs_cache_buf),
      0);
     

 printf("mount res: %i\n", res);
}

SpiffsBugReportFiles.zip Will appreciate any help or advice.

[pellepl]

Ah, you're using my spiflash driver. Then I can't really blame it on you can I? :) Will see things through.